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DESIGNING A NATIONAL ASSESSMENT SYSTEM: 
ASSESSING ABILITIES THAT CONNECT EDUCATION AND WORK 

Marcia Mentkowski 

Alverno College 
Milwaukee, Wisconsin 

ABSTRACT 

The purpose of this paper is to summarize whai we have learned from our study of relationships between 
abilities/outcomes learned in college and those abilities as they are performed at work. From these 
experiences and findings, we infer principles that should inform a national assessment system. Further, we 
draw implications, questions and issues that need to be considered if the recommendations we make for a 
national assessment system are to be realized. This paper contributes to the larger purpose of developing a 
process to assess higher order thinking and communication skills of college graduates in support of the 
National Goal of Literacy and Adult Learning. The overriding concern is that benefits accrue directly for 
faculty and students, employers and employees, and indirectly Tor policy makers in the public and private 
sector, as well as the public at large. 

This paper deals with issues of what to assess, and how to assess abilities that link education and work. While 
the primary interest in and locus for assessment is with the college student, upon entry, during college, and at 
graduation, the consequent questions that arise from this information will immediately redirect the public 
interest to how college learned abilities connect to work and citizenship. 

This paper argues that abilities can be assessed in ways that connect education and work, provided that 
abilities are defined in ways that make sense to both educators and students, employers and employees. 
Abilities that are defined with multiple components, as integrated, developmental and transferable, can be 
assessed within appropriate roles and contexts. Thinking critically, communicating effectively, and solving 
problems are abilities common to college education and work. Effective performance at work is made up both 
of intellectual and interpersonal abilities; thus, performance that integrates interpersonal and intellectual 
abilities should be assessed. Those abilities that enable education- work relationships, that is, that enable the 
transfer of learning from college to work and from work to further learning need to be assessed as well; one 
example is learning to iearn or self-sustained learning. 

Further, this paper argues that abilities can be assessed in ways that enable judgments of graduates* 
effectiveness, and in ways *liat help determine the effectiveness of college learning for work. We will need to 
identify abilities involved in "effective" work and citizenship roles, but information can be collected directly 
from college /aduates in work and service role. When abilities are defined and assessed in ways that 
connect education and work, then faculty and students, employers and employees can use the information both 
to improve instruction or training, respectively, and to determine how they arc meeting their own and others* 
expectations for learning outcomes and for work performance. Both an accountability and improvement 
agenda should be met within the frame system. 

The discussion of the above issues moves from principles learned from our research, that ultimately lead to a 
series of recommendations, followed by implications, issues and questions. A map that guides the reader 
throughout is Figure 1, pp. 4-6. These recommendations set forth essential components, principles, and 
strategies for a national assessment system that ultimately would contribute to the improvement of both higher 
education and life after graduation for the American citizen at work and in other roles in society. 
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DESIGNING A NATIONAL ASSESSMENT SYSTEM: 
ASSESSING ABILITIES THAT CONNECT EDUCATION AND WORK 1 

INTRODUCTION 

Assumptions Underlying this Paper 

There are several assumptions underlying this paper. First, we assume that the context for relating education 
and work includes before, during and after college. College population demographics show that the majority 
of college students no longer consist of 18-year-olds who begin college directly from high school, complete 
their education in four years, and then either go on to graduate school or enter the workforce. Rather, college 
students arc likely to be from a broad age range, with a heterogeneous work experience. Many combine 
college with full-time or part-time employment; our own students work before, during, and after college. 
Further, many postsecondary programs in technical schools, community colleges, and liberal arts colleges who 
preoare students fc~ professional careers have well-developed internship and field experience programs that 
call for assessment of student performance in field settings. 

Second, as workplace demands become more complex, employers and the professions are putting more and 
more resources into education, training, and continuing education programs. Educators in corporations and the 
professions are expecting more support in this effort from colleges and universities. Ilius, more persons will 
be in college across their lifespan. With so many college students wonting before, during, and after college, 
and so many workers requiring more learning opportunities, we need to see college learning in the context of 
work, and work in the context of college and other kinds of work-related learning. While this paper often 
uses language implying that transfer "from college to work" is me only interest, we assume that work 
influences college as much as college influences work. 

A third assumption is that college and university students and faculty, as well as employers and employees in 
business, community organizations, and the professions, are joined together in a mutual enterprise. All these 
persons are invested to some degree in improving Laming opportunities at college and in the workplace. 
They have a common interest in assessing abilities that prepare one for twenty-first century demands. These 
abilities and demands include participating in a global economy and global citizenship. The assumption is that 
a national assessment system can call on and count on this common interest. 

A fourth assumption is that workplace and citizenship demands are becoming so complex that, contrary to past 
perceptions, college is now a place where students are learning abilities that will be important for day-to-day 
work as well as for personal or leadership goals. The kinds of complex thinking, communication, and problem 
solving skills learned in college are in great demand in the workplace as well as one's personal life. 
Interpersonal abilities, leadership qualities, and self-directed learning skills are essential across all of these 
situations, particularly as business and industry moves to more of a "total quality management" environment 
(DcWitt, 1991). 

Fifth, both college faculty and employers have a joint interest in individuals meeting expectations for 
citizenship, and they assume that abilities developed in college and performed at work are ones that transfer to 



This paper was written in relationship to another paper commission '.u by NCES in order to enable the authors to focus 
on two separate areas identified by the Center, and at the same t-'mc to maximize the space allotted and to respect out- 
readers' patience. In collaborating in writing the papers, we aimed to establish logical relationships among the principles, 
recommendations, issues, and questions we set forth for a national assessment system. This paper's companion piece, 
"Developing a National Assessment System: Alvemo's Institutional Perspective," by Georgiue Loacker, sets context for 
this one, and should, if possible, be read first (although the papers can be read independently). Many of the principles 
learned and consequent recommendations in this paper assume those developed in Loacker's paper. Further, this paper 
has been framed, in part, within the context of Alvemo's ability-based, performance-assessment curriculum, it is best 
understood with that context in mind. 
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service and citizenship roles. Information from assessment can assist both faculty and employers to determine 
the degree to which graduates and employees are meeting their own expectations, those of faculty and 
employers, and of those roles outside of work. 

A sixth assumption is that any national assessment system will consider that abilities, assessed in context, will 
change, because the demands for performance and the contexts in which abilities are performed, will change. 

A seventh assumption is that the primary interest in and locus for assessment is with the college student, upon 
entry, during college, and at graduation. Likewise, we assume, that information from assessments of college 
learning outcomes at graduation will leave unanswered the question that students, educators, employers and 
policymakers are often most interested in: "Does college learning transfer *o wort; and service, and does 
college prepare individuals not only for entry level positions and beginning roles, but also for future positions 
and roles, which may be as yet undefined or unimagined? How do we know which abilities last a lifetime, 
which are needed for the twenty-first century, and which of these should be developed during college?" 
Thus, a national assessment system should assess and research how abilities are actually performed in the 
workplace to respond to societal expectations that higher education is to some degree responsible not only for 
immediate but future application of college-learned abilities, and to respond to higher education's expectations 
that one cannot prepare students adequately unless one can to seme degree project future needs. Thus, while 
tne primary investment is in the assessment of the college student, the consequent questions that arise from 
this information will immediately redirect the interest to how college learned abilities connect to work and 
citizenship afterward. 

Assumptions eight and nine relate to a national assessment system. We assume that readers accept the idea of 
a national assessment system rather than a single national test, and that they are persons in the process of 
considering what it means to assess the abilities of students during and after college. 

Finally, this paper shares an assumption with that of other papers commissioned for this project. Each author 
is commissioned to widen the lens, to review all the elements we believe should be part of a national 
assessment system. While we should identify implications, issues, and questions that flow from our 
recommendations, we should not prematurely impose limitations or feasibility criteria that could limit either a 
national vision or a set of national opportunities. Explicit concern for all the difficulties of implementation is 
set aside in favor of broadening the scope and potential of a national assessment system. 

Purpose 

The purpose of this paper is to examine whether abilities taught, learned, and assessed in college can form the 
basis for effective performance after college. If the conclusion is positive, then a national assessment system 
should measure how college graduates are performing those abilities that make for effective performance in 
work and citizenship roles afterward. 

Issues that Frame this Paper 

What Abilities/Outcomes Should be Assessed? 

The first of several broad issues that frame this paper is, "What abilities/outcomes should be assessed? 11 Can 
we clarify common elements of abilities identified in the national goals both in college and work? At least 
two conditions determine whether we can agree on these abilities as college educators and employers. One is 
the extent to which a national assessment system can gradually demonstrate that these college-learned abilities 
ire linked to effective performance in the world of work. Another is the possibility of defining the abilities 
through assessment criteria so that, while they transfer across contexts, they are assessed in context. 



DESIGNING A NATIONAL ASSESSMENT SYSTEM: ASSESSING ABILITIES THAT CONNECT EDUCATION AND WORK 



page 4 



How Should Abilities be Defined? 

A related issue is "How should abilities be defined?" A significant section of this paper deals with this issue 
because definition is critical. It helps us judge whether additional abilities should be incorporated into those 
identified as national goals. Further, some abilities to enable transfer to the world of work may have to be 
learned in college. 

Arc Abilities Learned in College? 

A third issue is whether we can demonstrate that these abilities are learned in college as a result of the 
curriculum. This paper deals with this issue in relation to the arguments and evidence provided in its 
companion commissioned paper (Loacker, 1991) (See Footnote 1). In both papers, various educational 
assumptions underlie how abilities are defined. While the research reported is considered neither definitive 
nor representative of the field, evidence is cited primarily on Alverno graduates because they experienced an 
ability-based curriculum, including performance assessment, and we have research evidence linking student 
changes in abilities to this curriculum. Thus Alverno graduates' performance at work is of interest because we 
have evidence that they learned some of these abilities in college. The direct links between graduate 
performance at graduation and five years later is still under investigation in the Alvemo longitudinal study. 

How and When Should We Assess? 

A fourth issue is how and when to assess for these abilities. These abilities should be assessed before, during, 
and at the endpoint of the college experience. Ideally, a complete national assessment system would also 
assess for these abilities after college in the workplace for two reasons. First, educators and policymakers are 
interested to see if graduates who learned these abilities in college are performing effectively in a competitive 
workforce and, as citizens, in society in general. Second, the study of graduates' abilities in the workplace 
and in service roles can illuminate the definitions of abilities that should form the basis for what is taught and 
assessed in college, and so contribute to higher education's ability to connect education and work. 

WHAT HAVE WE LEARNED? 

We discuss these issues by identifying what we have learned from a series of studies that examine 
relationships between college outcomes and work. Our folio wup research, assessing both our graduates' 
performance of work and the performance of outstanding professionals who are not our graduates, supports our 
conclusions about what abilities to assess and how to assess for them in college students. From our findings, 
we make recommendations for the design of a national assessment system. Throughout, we identify 
implications, issues, and questions that flow from recommendations for designing a national assessment system 
that can assess abilities learned in college and that transfer to work and citizenship. Following is a complete 
list of "What Have We Learned" statements. Each is discussed in turn, with recommendations made for 
designing a national assessment system, and consequent implications, issues, and questions (See Figure 1). 

1. Abilities can be defined in ways that connect education and work. These can be assessed within 
professional roles in appropriate contexts, as well as in college, in order to achieve both accountability and 
improvement. 

2. Abilities defined as having multiple components, and as integrated, developmental and transferable, are 
likely to make sense both to educators and employers. 

3. Thinking critically, communicating effectively and solving problems are abilities common to college 
education and work. Effective performance at work is integrated; it is made up both of intellectual and 
interpersonal abilities. 
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4. To effectively transfer college-learned abilities, students need to develop learning to learn skills, or self- 
sustained learning. Assessment that incorporates feedback and opportunities to self-assess fosters self- 
sustained learning. 

5. Comparing faculty-defined abilities to those demonstrated by outstanding professionals enables faculty to 
identify abilities students need for particular professions. 

6. Complex abilities that connect education and work— including self-sustained learning-can be assessed in 
graduates' work performance in a variety of professional contexts. Some abilities can be linked to college 
learning, and some distinguish effective performance at work. 

7. Faculty, professionals, and employers will invest in understanding the relationship between education and 
work if they can create contextually rich descriptions of performance in relation to their judgment of what 
abilities to develop. 



Using the Map 

We suggest using Figure 1 on pages 6, 7 and 8 as a "map" to assist in navigating the following text. 



Figure 1. Designing a National Assessment System: Assessing Abilities That Connect Education and Work 



WHAT HAVE WE LEARNED? 


RECOMMENDATIONS 


IMPLICATIONS 


ISSUES AND QUESTIONS 


1. Abilities can be defined in ways that 
conned education and work. These 
can be assessed within professional 
roles in appropriate contexts, as well 
as in college, in order to achieve 
both accountability and 
improvement. 


1. A national assessment system 
should assess for abilities that 
connect education and work. 
These should be assessed on 
the job within appropriate 
contexts, as well as in the 
college setting. 


1. Some abilities assessed by educators 
in college and employers at work will 
not be measured by a national 
assessment system that looks at 
graduates' performance on the job. 

1. Assessment techniques may differ in 
education and work contexts, but 
common abilities could cross both. 


1. What abilities should be 
assessed? 

1 . What abilities are learned in 
college and performed at work? 


2. Abilities defined as having multiple 
components, and as integrated, 
developmental and transferable, are 
likely to make sense both to 
educators and employers. 


2. A national assessment system 
should define abilities as having 
multiple components, and as 
integrated, developmental and 
transferable, so they make 
sense to educators and 
employers. 


2. Assessments will measure cognitive 
and affective, knowledge and ability 
components simultaneously. 

2. Measuring changes in abilities from 
college to post-college may call for 
qualitative measurement and 
nonadditive, nonlinear change 
model" 


2. How are oilities best defined? 

2. What does it mean to assess 
both cognitive and affective 
components simultaneously? 

2. How do we assess content and 
abilities, knowledge as well as 
performance? 

2. How define construct validity 
when abilities are 
multidimensional rather than 
unitary constructs? 

2. What is "good" evidence? 

2. What is the oevelopmental 
sequence of abilities? 

2. Do abilities become integrated 
in performance? 

2. How measure changes in 
abilities after college? 

2. How define reliability when we 
assume change rather than 
consistency? 

2. Can a{J outcomes be specified? 
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effectively and solving problems are 
abilities common to college 
education and work. Effective 
performance at work is integrated; it 
Is made up both of intellectual and 
interpersonal abilities. 
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should assess for abilities 
already identified in the national 
goals and should provide for 
performances that integrate 
intellectual and interpersonal 
abilities. 


Assessments win measure intellectual 
tt nd interpersonal abilities. 


3. What does ft mean to assess 
intellectual and interpersonal 
abilities simultaneously? 

3. How assess integrated 
performance? 
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Figure 1 (continued). Designing a National Assessment System: Assessing Abilities That Connect Education and Work 
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WHAT HAVE WE LEARNED? 



To effectively transfer college- 
learned abilities, students need to 
develop learning to learn skills, or 
self-sustained learning. Assessment 
that incorporates feedback and 
opportunities to self-assess fosters 
self-sustained learning. 



Comparing faculty-defined abilities 
to those demonstrated by 
outstanding professionals enables 
faculty to identify abilities students 
need for particular professions. 



RECOMMENDATIONS 



4. 



A nationa* assessment system 
should assess self-sustained 
learning and should include 
feedback and opportunities to self- 
assess. 



5. The method that some educational 
institutions and consulting firms 
use to measure abilities in the 
workplace to improve student and 
employee performance can be 
used as a strategy in a national 
assessment system. 



IMPLICATIONS 



Measures will include open- 
ended, indepth, interactive, 
sustained interviews of 
graduates' perspectives. 



This method could be used to 
assess abilities after college in 
context, as well as for identifying 
abilities. 



ISSUES AND QUESTIONS 



4. How assess self-sustained 
learning? Is it feasible? 

4. How assess for integrated 
performance? 

4. How give feedback and 
opportunities to self-assess? 



5. Can the method assess abilities in 
college and at work? 

5. How sample faculty/institutes to 
build the research base for 
creating abilities? 

5. How feasible is it to follow 
graduates after college? 

5. How feasible is it to assess for 
graduates' performance at work? 

5. What are the barriers to using 
information about graduate's 
performance at work back on 
campus? 

5. How can performance at work be 
linked to college instruction? 

5. How assess for performance-in- 
context? 
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Figure 1 (continued). Designing a National Assessment System: Assessing Abilities That Connect Education and Work 



WHAT HAV* WE LEARNED? 



6. Complex abilities that connect 
education and work-including sett- 
sustained learning-can be 
assessed in graduates' work 
performance in a variety of 
professional contexts. Some abilities 
can be linked to college learning, 
and some distinguish effective 
performance at work. 



7. 



Faculty, professionals, and 
employers will invest in 
understanding the relationship 
between education and work if they 
can create contextually rich 
descriptions of performance in 
relation to their judgment of what 
abilities to develop. 



RECOMMENDATIONS 



A national assessment system 
should sample graduates' 
performance at work and in 
citizenship roles, as well as in 
college, and judge performance 
in relation to those 
developmental criteria that 
distinguish effective 
performance. 



7. Faculty, employers, and public 
representatives should be involved 
in making judgments about the 
abilities essential to effective post- 
college performance at work and 
in citizenship roles. 



IMPLICATIONS 



6. Institutions, employers, and 
alumnae will need to work 
together to design and develop 
appropriate strategies for 
sampling performance through 
work samples, portfolios, 
behavioral event interviews, 
critical incident techniques, or 
other assessment technologies 
already in place at work. 

6. We will need to identify abilities 
involved in "effective" work and 
citizenship roles, but information 
can be collected directly from 
college graduates at work and in 
service. 



7. Creating a context for 

participation is as important as 
developing assessment 
techniques. 



ISSUES AND QUESTIONS 



6. How assess integrated 
performance? 

6. How assess performance-in- 
context? 

6. How account for differences in 
work settings and in professional 
roles? 

6. How use expert judgment to code 
abilities in performance? 

6. How validate expert judgment of 
complex, integrated performance? 

6. How judge effectiveness? 

6. How make assessment cost 
effective? What are legal issues 
involved in performance samples 
of employees for post-college 
studies? 



How assess for what employers 
and the public really want to see: 
Personal maturity, leadership, 
ethical decision-making, 
integrated performance? 
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Principle Learned #1, Abilities can be defined in ways that connect education and work. These can be 
assessed within professional roles in appropriate contexts, as well as in college, in order to achieve both 
accountability and improvement. 



Abilities identified in the national goals have elements that appear in studies linking abilities learned and 
assessed in college with those demonstrated at work. Those abilities that do connect can be assessed both in 
college to measure the outcomes of college learning and in the workplace to demonstrate accountability and to 
improve definitions of abilities back on campus. 

Figure 2 illustrates an example of abilities that connect college and work drawn from National Goal #5 and 
Alverno research. Figure 2 also means to convey that academic-specific knowledge, procedures, and abilities 
contribute to performance of abilities in coVege. Further, context and/or task-specific knowledge, procedures, 
and orientations contribute to the way abilities are performed at work. 



Figure 2. Assess Abilities that Connect Education and Work. 



COLLEGE ONLY 


ABILITIES THAT CONNECT COLLEGE AND WORK 


WORK ONLY 




• Think critically 

• Communicate effectively 

• Solve problems 

• Use interpersonal abilities 

• Use self-sustained learning 




Assess abilities using 
performance assessment, 
portfolios, performance 
interviews 


Assess abilities using performance 
interviews, portfolios, etc. 


Assess abilities using 
work samples, 
performance appraisals, 
assessment center 
method 


T 

Academic-specific 
knowledge, procedures, 
orientations 




t 

Work-specific 
knowledge, procedures, 
orientations 



A variety of performance assessment methods can be used both in college and at work. For instance, there an 
examples of the use of an assessment center method at work, for development as well as selection. Alverno 
educators have adapted the method to use it for individual development. Educators also use portfolios and 
performance interviews assessing performance during college. 
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Abilities That Have Been Defined 



Abilities can be identified that enable educators to connect an undergraduate liberal ans education to 
professional performance. Appendices A, B. and C identify several sets of abilities/outcomes that are a 
reference base for this paper. Appendix A lists abilities that Alvemo College has constructed during 18 years 
of experience in teaching and assessing abilities for credit leading to graduation. These are defined as 
essential outcomes of college that will serve Alvemo graduates throughout their lifetime (Appendix A). 
These abilities continue to be refined and researched. 



In addition to the abilities in Appendix A, Appendix B lists abilities currently under study in five-year 
alumnae performance. These are faculty-identified outcomes that all students are expected to achieve, but that 
arc not assessed for credit leading to graduation. They include a set of Six Performance Characteristics 
(Integration, Independence, Creativity, Awareness, Commitment, and Habituality). They also include learning 
to learn, self-directed or self-sustained learning; self-assessment; cooperation/collaboration in community 
activity (academic, civic, professional); and understanding oneself in relationship to the world of v/ork/civic 
duty and to a multicultural context. There are other outcomes that are developed according to the special 
characteristics of each discipline/profession, reflecting the unique style, interests, and abilities of each student. 

In addition, Appendix B lists professional competences. The College has worked to research abilities of 
outstanding professionals who are not our graduates which can guide, shape, evaluate, and validate faculty 
definitions of abilities/outcomes in the disciplines and professions. This work was funded in part by the 
National Institute of Education and a Title III grant from the Strengthening Developing Institutions Program. 
They include definitions of teacher competences (Diez, 1990), and nursing competences (DeBack & 
Mcntkowski, 1986). They also include management competences studied by McBer and Company (Boyatzis, 
1982; Evarts, 1982) and Alverno. Alverno studied a wide range of professionals in the public and private 
sector (Schall, Guinn, Qualich, Kramp, Schmitz & Stewart, 1984) and women managers and executives in the 
private sector (Mentkowski, O'Brien, McEachem & Fowler, 1982). The Klemp and McBer sets of abilities 
are listed in Appendix B, because these generally are defined and researched in ways that are symbiotic to the 
definition of ability or competence Alvemo faculty use, and were generated through methods Alvemo 
researchers also use. The Klemp and McBer competences were included in the synthesis Alvemo created to 
code performances of five-year alumnae after college (Rogers & Talbott, 1991). A short form of this 
codebook is found in Appendix C, which is a synthesis of the abilities in Appendices A and B. Finally, two 
additional sets of competences are listed in Appendix B. George Klemp (1988) synthesized competences from 
studies (1977, 1982, 1991) at Cambria Consulting, and Lyle Spencer and Signe Spencer (1986, June) 
synthesized competences from McBer and Company research. To date, McBer has completed over 750 job 
competence studies (McClelland, Spencer & Spencer, 1990). 

These "lists" are referenced here in order to make the point that abilities can be identified that enable 
educators to connect an undergraduate liberal arts education to professional performance. 

Approaches to Relating Student, Alumnae, and Other Professional Abilities/Outcomes 

to the World of Work 



We used four approaches to examine the relationship between the outcomes shown by college students and the 
world of work. Four independent data sources result: (a) longitudinal studies of student and alumnae outcomes 
(Mentkowski, 1988; Mentkowski & Doherty, 1983; 1983, revised 1984; 1984; Mentkowski & Strait, 1983; 
Mentkowski, Rogers, Deemer, Ben-Ur, Reisetter, Rickards, & Talbott, 1991); (b) studies of two-year and five- 
year alumnae perceptions of the abilities involved in the work place, and of the value of learning in their own 
evolving life goals (Deemer & Mentkowski, 1990, in progress; Mentkowski, Much & Giencke-Holl, 1983; 
Giencke-Holl, Mentkowski, Much, Mertens & Rogers, 1985); (c) studies of the abilities actually used by other 
professionals in job situations (DeBack & Mentkowski, 1986; Mentkowski, et al., 1983; Schall, et al., 1984), 
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and (d) studies of the abilities actually used by five-year Alverno alumnae and judged for effectiveness 
(Mentkowski, et al., 1991; Rogers & Talbott, in press). 2 

The data from alumnae and other professionals who arc not Alvemo alumnae suggest that graduates 
experience direct transferability, into the workplace, of intellectual and interpersonal abilities learned in 
college, and there are key abilities exercised by effective professionals which are statistically related to college 
learning and conceptually linked to the abilities identified, taught and assessed by Alverno faculty. Further, 
alumnae continue as self-sustaining learners, showing "learning to learn" skills that enable them to develop and 
adapt abilities, and to achieve job and career satisfactions. Finally, five-year alumnae perform abilities at 
work. Initial results suggest that effective alumnae performance is distinguished by taking achievement actions 
and having a positive influence within a broad organizational context including the development of others 
(Mentkowski, et al., 1991; Rogers & Talbott, in press). 

Finding: Abilities Function as an Organizing Principle 
for Role Performance and Career Satisfaction 

Abilities Structure Performance at Work 

What are the consequences of organizing college learning nnd assessment around abilities? Although alumnae 
used abilities taught in the curriculum, they also used abilities to create a theory of action that gets tested out 
in various work situations. This sharply contrasts to the view that technical knowledge alone is the basis for 
effective performance. Alumnae said they do learn new technical skills, but they did not emphasize this 
knowledge when they deicribed how they go about deciding what to do ("My job is never the same...I use 
communication and analysis because you work very independently...you have to analyze the financial 
statements from taxpayers and determine a course of action"). 



Pau sources. Results are reported from a) curriculum-embedded performance assessments, b) college designed instruments and 
interviews and c) a battery of 12 external measures of generic abilities, learning styles, and moral, intellectual and ego development 
(human potential measures). These were completed longitudinally on three occasions (1976/1977; 1978/1979; 1980/1981) by the 
entire entering classes of 1976 and 1977 during college (n=706), and most measures were completed again on a fourth occasion five 
years later (1986/87). Measures of abilities, learning styles, motivation, cognitive, moral and ego development were employed along 
with in-depth, confidential interviews, surveys of student perceptions and background characteristics, and behavioral event interviews 
(McClelland, 1978) of alumnae performance across settings in work, personal life, and service. Student participation rates ranged 
from 84 to 99 percent: alumnae (n=358) rates ranged form 59 to 88 percent. Data from curriculum-embedded performance 
assessments in the curriculum, academic reports and a faculty rating of performance characteristics with background factors 
controlled, were related to changes on the battery of external measures using multiple linear regression, ANOVA for repeated 
measures and path analysis. Interviews were coded via ethnographic and thematic analysis. 

More specifically, the battery of 12 human potential measures and college designed instruments were administered to two complete 
entering classes and one graduating class (altogether about 750 students). A subsample (n=80) completed in-depth interviews as well 
at the end of each year in college. Hie entering classes completed the s*me battery two years after entrance and again two yean 
later, near graduation, and again five yean later. Thus, we have a set of longitudinal results that can be double-checked against 
results from a cross-sectional study of 60 graduating senion who participated in 1978 as senion and 1980 as alumnae, who were 
compared with entering students who later graduated (controlling for retesting and attrition, with initial selection facton, such as 
disposition to change, probably uncontrolled). The data on students who completed die 12 external instruments on three occasions 
provide a parallel stream of longitudinal information alongside these same students' progressive performance on five college-designed 
measures. The design includes two age cohorts (age 17 to 19 and age 20 to 55 at entrance), to examine die effects of maturation, 
and two achievement cohorts (high and low, based on number of consecutive assessments completed in die curriculum) to examine 
the effects of performance in die curriculum. Two class cohorts, with die second cohort analyzed for weekday venus weekend imr 
frames, further enhance representativeness, although only further longitudinal cohorts could truly control for curriculum and societal 
change effects. The time series design holds time constant and allows performance in die curriculum to vary, so we can attribute 
change to performance in die curriculum in die absence of a control group of students who did not attend Alvemo. As mentioned, 
we also control for several age, background, and program variables as well as pretest scores when we study die effects of 
performance in die curriculum. In studies of current students, their current portfolio* and other curriculum performance assessments 
are judged on dimensions of performance by expert judges and related to abiliue. that define the major. 
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Analysis of work behavior of experienced, outstanding professionals who are not Alverno alumnae showed 
that professionals used the abilities, and that abilities were an organizing principle for role performance. In 
managers' performance interviews, we coded few examples of using specialized knowledge as a basis for 
performance. Managers used these skills, but when they described what they do in sin uiions, broad abilities 
formed the basis for their actions. In contrast, performance interviews from five-year alumnae were coded for 
more examples of using specialized knowledge. Thus knowledge was important, but knowledge alone did not 
make for effective performance. 

Technical Skills Are Not Enough 

The fact that professionals across all our studies demonstrated a wide range of complex abilities implies that 
graduates with functional or technical skills alone, or with knowledge alone, will not be effectively prepared to 
meet the demands of either nursing or management positions. It also implied that abilities that cross position 
levels and even careers can be abstracted by colleges and built into general education curricula. Abilities that 
are profession-specific (e.g., "entrepreneurial abilities" for managers or "coaching" for nurses) can become the 
cornerstone for further development in majors. 

Certainly specific training is needed for any entry level position, but for persons who plan a career in the two 
professional areas we researched, an education to prepare them for the future needs to include learning to 
integrate a number of abilities, to test them out in a range of actual work situations, and to critically appraise 
one's own performance. 

Alumnae Experience Competence 
and Career Satisfaction 

The experience of competence is a key factor in career management and job change. Graduates viewed work 
not just as a job, but as a career that changes and develops through work experience. Older women had a 
specific direction toward long range career goals; younger ones were more immediately focused on developing 
competence in their present job. But for all of them, career satisfaction was strongly related to their 
experience of competence on the job. 

Experiencing competence seemed to be a critical factor in whether graduates changed their job or career; it 
also seemed to "carry over" from one job to another. While graduates adapted to problems encountered in the 
first two years of work, including the traditional "reality shock" or disillusionment experienced by most new 
graduates, the persistent feeling of not being able to perform in the job role, for whatever reason, led to 
change of jobs, if not career fields. Such changes were generally successful and appeared to reestablish the 
graduates' feeling of professional competence, the basis for their self-esteem. Apparently, these graduates 
could change jobs and careers effectively if they have a sense of competence and strive to develop it in 
whatever role they choose. 

Using abilities led to self-confidence on the job and was the basis for job satisfaction. Being able to perform 
abilities led to staying on the job. 



PERFORMING ABILITIES -> SELF CONFIDENCE -> JOB SATISFACTION 
LACK OF OPPORTUNITY TO PERFORM ABILITIES -» JOB OR CAREER CHANGE 
CONTINUED LEARNING -» DEVELOPING AND ADAPTING ABILITIES -> JOB SATISFACTION 



IS 
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Another important indicator of job satisfaction was the degree to which alumnae experienced continued 
learning, an intrinsic value which motivated not only career development and jol uioice, but also determined 
whether an alumna continues to develop and adapt her abilities. 

Alumnae Perform Abilities at Work 

As demonstrated in other studies, the performance interview data showed thabfive-year alumnae frequently 
accurately self-assessed a good performance as good. Also, they sometimes spontane jusly abstracted their key 
abilities in describing events or used their self-assessment to evaluate whether their abilities were sufficient for 
what was required. Of course alumnae were not immune to occasionally overlooking a weakness or mistake 
but generally they more or less accurately self-assessed their performance. This relates to the expectation of 
Alverno faculty that students transfer their ability to self-assess college work, to self-assess their performances 
after college. The association of accurate self-assessment with positive evaluations of performance during an 
event suggests that self-assessment may improve ongoing performance, as well as future performance. Self- 
sustained learning may often be an important part of the performance itself. 

Alverno alumnae also often gave their colleagues or staff at work specific feedback and resources that improve 
their performance. Not only does this development of others demonstrate internalization of the value of 
learning and taking on broader commitments, but it also should encourage college faculty to envision 
educational effects beyond those that they directly teach. This finding suggests an important link: when 
Alverno faculty give students feedback and self-assessment opportunities in relation to performance, then those 
students who have internalized educational principles, themselves became effective staff developers in the 
work force. Apparently, experience with the performance-assessment process had effects beyond its 
immediate impact on students or alumnae. 

We conclude from these findings that (1) abilities connect education and work in both the performance of 
graduates from an ability-based curriculum and the performance of outstanding professionals who graduated 
from other programs; and (2) abilities function as an organizing principle for role performance and career 
satisfaction. Therefore, we make the following recommendation. 



Recommendation #1. A national assessment system should 
assess for abilities that connect education and work. These 
should be assessed on the job within appropriate contexts as 
well as in the college setting. 



IMPLICATIONS, ISSUES, AND QUESTIONS 

• What abilities should be assessed? 

An initial response is that those that connect education and work. But if we assess only for abilities that 
connect college and work, then some abilities assessed in college and at work— namely, those that are not 
demonstrated in one of the contexts— will not be measured by a national assessment system that looks at 
graduates' performance on the job. But assessing both sets of abilities, those that do cross settings and those 
that do not, would help us identify those abilities that should be taught in college and those abilities that are 
also performed at work. Those abilities that do connect can be assessed both in college to measure the 
outcomes of college learning and in the workplace to demonstrate accountability and to improve definitions of 
abilities back on campus. 
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• What abilities are learned in college and performed at work? 

A second question is "Can we clarify elements of abilities that are learned in college and performed at work?" 
We take up this issue later ("What Have We Learned," #6). Before wp ran move on to a discussion of the 
issues that assessment of abilities at work raise for a national assessment system, we must first deal with the 
question of how abilities are best defined. This is the focus of the next section. 



Principle Learned #2. Abilities defined as having multiple components, and as integrated, developmental 
and transferable, are likely to make sense both to educators and employers. 



Defining Abilities 

Many institutions are making outcomes explicit by defining abilities like critical thinking that they expect 
students to master by the end of general education or the major. But how are they defining these abilities? 
How do employers define abilities for purposes of training on the J )b and performance appraisal? 

Definitions of critical thinking and other complex abilities are informed by new definitions of intelligence. 
Howard Gardner's (1983) multiple frames of mind and Robert Sternberg's (1986) triarchic theories of 
intelligence describe the complexities of critical thinking. Further, even critical thinking measured by a 
traditional instrument like the Watson-Glaser Critical Thinking Appraisal (1964) shows a pattern of 
development during college and afterward that differs from the pattern on a measure like the Test of Cognitive 
Development (Renner. Fuller, Lochhead, John, & Tomlinson-Keasey, 1976) in Alverno longitudinal studies 
(Mentkowski, et al., 1991). 

Thus, defining abilities is not an easy task; how they are defired remains a critical issue. The definitions 
specified below undergird all the recommendations made in this paper for assessing abilities in a national 
assessment system. 

Abilities have multiple components, including motives or dispositions, self-perceptions and attitudes, skills, 
behaviors, and knowledge. This definition of ability undergi :.ds the research reported in this paper, whether 
conducted by Alverno researchers (DeBack & Mentko* 'ki, 1986; Mentkowski, et al., 1982; 1991; Rogers & 
Talbott, in progress), or by other faculty and researchers i the United States and the United Kingdom (Burke, 
1989; Jessup, 1991; Klemp, 1988; McClelland, et al., 1990). Clearly, knowledge is only one component of an 
ability and abilities include skills and behaviors. Thus, abilities link knowing and doing, knowledge and 
performance. These components differ in the degree to which they are directly observed in performance. 
Some are observed with confidence. In any event, ability components serve as a heuristic in defining abilities. 

The terms ability and competence are somewhat synonymous. Alverno faculty have moved to the term ability 
so as not to be confused with competencies that are often narrowly defined as a list of behaviors. Those 
researchers we have worked with share the definition of competence that invokes a causal relationship to 
effective performance, as Klemp and Sokol describe below: 

The kernel of competence lies in effective performance by the individual. For the purpose of this report, 
a competency, or component of overall competence, is a characteristic of an individual that underlies 
effective work performance. A competency can be any human quality: It can be knowledge, a category 
of usable information organized around a specific content area (for example, knowledge of mathematics); 
it can be a skill, the ability to demonstrate a set of behaviors or processes related to a performance goal 
(for example, logical thinking); it can be a trait, a consistent way of responding to an equivalent set of 
stimuli (for example, initiative); it can be self-schema, a person's image of self and his or her evaluation 
of that image (for example, self-image as a professional); or it can be a motive, a recurrent concern for a 
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goal state or condition which drives, selects, and directs behavior of the individual (for example, the need 
for efficacy). A person may possess many of these characteristics, but by our definition, if the 
knowledge, skill, trait, self-schema, or motive is not explicitly related to effective performance, it is not a 
competency (Klemp & Sokol, 1980, p. 2). 

Alverno faculty define abilities as integrated, developmental, and transferable (Alvemo College Faculty, 1979 
revised 1985). Abilities are integrated, inseparable parts of a person, not a set of discrete skills. Our research 
shows that integration can occur among several abilities within a situation or a context (e.g. intellectual and 
interpersonal; intellectual and socio-emotional). Integration can occur between knowing and doing (abilities 
and use; thought and action, knowledge and performance). Integration can occur between perceptions of what 
abilities are needed and actual abilities performed (Mentkowski, 1988). Dearly, the question of whether 
abilities become integrated is critical. Does an individual actively construct abilities into a complex whole and 
show coordinated use of multiple abilities in a situation, in a way that the performance of each ability is 
supported by the others? 

Alverno faculty also define abilities as developmental, or teachable. If an ability is developmental, it can be 
defined in sequential descriptions or pedagogical, cumulative levels (see Appendix A for abilities defined as 
performance levels by Alverno faculty) that describe increasingly complex elements or processes for teaching 
and assessing performance. Thus, abilities are often defined via criteria that are increasingly complex across 
pedagogical levels or professional positions (McClelland, ct al., 1990; see Appendix D). 

Finally, Alverno faculty define abilities as transferable. They do not see them as a set of discrete skills 
performed and dictated solely by a particular setting or context. In our experience, educators identify with 
such a definition because they need to teach and assess for abilities that prepare students for the many roles 
and settings in which graduates will perform. Transferability is critical because even experienced professionals 
continue to change positions, roles, and responsibilities. While a broadly-defined ability does not usually have 
a one-to-one correspondence with observable actions in performing a professional role, it represents underlying 
characteristics that can be applied to describe the successful integration of a variety of ability components. 

Our studies confirm that complex abilities can be successfully integrated as a result of a college experience 
stressing an ability-based, outcome-centered approach to learning (Mentkowski, 1988; Mentkowski & Doherty, 
1983, revised 1984; 1984). These abilities can be identified, developed, and used to achieve success in 
educational and work environments. We can also identify contributors to effective outcome -centered learning. 
These contributors include instruction itself, as measured by the amount of student change on faculty-designed 
instruments and those from outside the college, and other curricular elements, like feedback and self- 
assessment, identified by students as important for their learning. Our data also showed the impact of 
education on the demonstrated abilities of alumnae and other professionals. 

Finding: Abilities/Outcomes Are Complex, Integrated Human Abilities 

We researched outcomes through several different frameworks and measures in our longitudinal studies. 
Clearly, an ability like critical thinking has cognitive, affective, kinesthetic, and social dimensions. It is 
expected to transfer to multiple aspects of work and community life after college. It is clear that definition 
and measurement of college outcomes needs to include a range of processes and dimensions: cognitive/ 
intellectual and affective/socio-emotional processes and perc ptions, motivation and performance dimensions. 
The outcomes studied by our battery of twelve external measures were differentiated into two separate factors 
at entrance to college and two years later, but were integrated by graduation. Educators may need to 
differentiate cognitive and affective aspects of abilities in order to teach for them, but they should assess them 
simultaneously. Most educators are struck by the difficulty of any attempt to separate these aspects. 

It is also clear from our study of student performance on external measures that educators defining abilities 
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needed to attend to individual differences in level of cognitive development. This has several implications for 
developing instruction and assessment. Faculty who rate students on a set of performance characteristics 
seemed to be tapping student development, since the ratings correlated with cognitive-developmental level If 
Alverno faculty were thus aware of students 1 developmental level—along with the more tangible abilities 
involved in their day-to-day assessments-we might infer that they also took developmental level into account 
in instructional planning, in their interactions with students, and in designing assessments. 

It is also clear that complex outcomes need to include a performance dimension. Learning to perform, to link 
knowledge to performance, is a dimension of abilities separate from the cognitive-intellectual one. This was 
borne out by the correlation of cognitive-developmental level with the concrete experience/abstract 
conceptualization dimension of learning styles. Cognitive-developmental level did not correlate with the 
reflective observation/active experimentation dimension. Educators have long sought an adequate learning 
theory that incorporates not only knowledge and cognitive/intellectual processes, but also the more practical 
learning that occurs when ideas are tested out in actual situations. This practical learning can be expected to 
transfer across contexts to the world of work. For students in our study, learning to perform, to link 
knowledge to performance, enabled them to find reasons for trying to learn in a variety of ways. They tried 
out the abilities through application to professional performance and in their personal life. By doing so, they 
cxperientially validated the abilities or competences they were developing. The concept of "competence," 
which implies knowledge and action, became a motivational link as well. Students began to see themselves as 
competent. Thus, outcomes showed a perceptual and motivational dimension that assist in their internalization 
and transfer. Values and motivation for performance had their roots in students 1 justification for learning as a 
stepping stone to a career and economic mobility. Along the way, self-sustained learning, a liberal arts value, 
became part of the student's reasons for continuing in college. The student came to perceive herself as a self- 
directed learner, who seeks "well-roundedness," as well as career goals. 

Complex abilities, which include cognitive, affective, behavioral, motivational, and perceptual components, did 
fit together and/or integrate to some degree by graduation. This suggests that the abilities are ultimately 
integrated in performance. 

Finding: Abilities/Outcomes Are Developmental and Teachable 

Abilities/Outcomes Develop as the Result of the Curriculum 

In our studies, complex outcomes or abilities changed over time and were related to performance in the 
learning process. Thus, they are developmental or teachable. We linked outcomes specifically to college 
instruction in six ways: (1) by relating students 1 change on faculty-designed assessments to instruction 
(Alvemo College Assessment Committee/Office of Research & Evaluation, 1980; Friedman, Mentkowski, 
Dcutsch, Shovar & Allen, 1982; Friedman, Mentkowski, Earley, Loacker & Diez, 1980), (2) by analyzing in- 
depth confidential semi-structured interviews of student constructions where students attributed changes in their 
learning to curricular elements, (3) by analyzing student change on twelve external instruments drawn from 
outside the college (Mentkowski & Strait, 1983; Mentkowski, 1988), (4) by analyzing alumna ratings and 
confidential interviews where graduates described using abilities in post-college settings (Giencke-Holl, et al., 
1985; Mentkowski, et al., 1983; Deemer & Mentkowski, in progress), (5) by relating educational background 
to the demonstrated abilities of professionals who are not Alverno alumnae (DeBack & Mentkowski, 1986; 
Mentkowski, et al., 1982) and (6) by studying alumna performance in work and personal roles (Mentkowski, 
et al., 1991; Rogers & Talbott, in press.) All of these sources validate the testimony of faculty who say they 
see students learning, of external assessors who credential some of these abilities, and of other students and 
alumnae who .say they are learning abilities, and demonstrate this by the increasing complexity of their 
descriptions through time and across settings. 
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Older Students Also Develop Outcomes 
as the Result of Instruction 

Student progress through the assessment process showed no noticeable deficit for the cider student. The very 
structure of Weekend College (an alternate time frame that requires more independent learning and that is 
attended mostly by adult students) presumes that the older adult can move at a more intense rate. Not only 
must she evidence this cognltively, but also in organizing multiple roles and responsibilities. Yet, there is no 
evidence of older students suffering any disadvantage from having been away from formal academic work. In 
fact, there was a cognitive advantage that allowed them not only to cope with the program but to cope with a 
concentrated program that is even more demanding. That presumption is borne out, by students attending 
Weekend College, in their performance on external instruments. Their performance changes were not fewer 
than those of students in the regular college time frame. In fact, our analyses of the external instruments show 
that age was initially an advantage in some areas because the experience that it implied enabled the student to 
take on this cognitive overload and deal with it successfully. Older students also developed their abilities as 
the result of instruction. By the time they were more than two years along, the educational environment itself 
was a more likely determinant of learning. At the same time, older students began at the same place as 
traditional age students in some areas, such as their understanding of classroom learning processes and roles. 

This difference between the older and younger adult showed up after college. The experienced adult showed 
more specific direction toward long-term career goals, in contrast to the younger graduate who was more 
focused on immediate ones. Both groups, however, reported having to apply the same kinds of abilities to 
post-college work settings— interpersonal and reasoning abilities. Thus, adults had an advantage in some areas 
but not in others. But it appears that the ability-based curriculum at least was capable of capitalizing on the 
differential abilities of the new learner. 

As context for this finding, it is important to no;e that our institution has a mission to serve working class 
students who are often first generation college students. For this student group, higher education can build on 
their particular strengths and background and enable them to continue capitalizing on it even after they have 
graduated. 

Abilities/Outcomes Develop at Different Times 

Equally important was the time frame for development. There were differences in when complex abilities 
develop during the college years. And as educators have always suspected, there was a difference between the 
general education experience and the later years when the student focused on a major. Older and younger 
students performed differently with respect to some cognitive-developmental patterns and abilities but not 
others. Further, abilities that may be differentiated during the first two years of college became integrated 
during the last two years, although how this happens is not clear to us now. The competence models 
developed from effective professionals in nursing and management show that abilities differed in complexity 
and sequence and suggest that the order of abilities is important and can be identified. Professionals were 
more likely develop some abilities exclusively on the job. Experience sometimes added to a student's ability 
to take advantage of college, but some key abilities critical for effective work performance were developed 
over time in long term formal learning experience. 

Alumnae Continue to Develop and Adapt Abilities 

We found that abilities were refined depending on their specific application. They were also combined in 
various ways given situational demands, and they were adapted to fit a particular action plan. Alumnae spoke 
again and again of combining ("trying to take everything and put it together in a workable process") and 
modifying their abilities ("I should not have come on so directly in getting him to try out this new equipment. 
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I needed to show him how we can get the work coordinated and out faster, first"). In the studies of 
professionals, effective managers and nurses consistently used a combination of abilities in a single situation. 
The same holds true for five-year alumnae performance studies. 

Developmental Patterns Are Subtle and Complex 

When we look at the rate and quality of change, we note the kinds of subtle and complex developmental 
patterns that will ultimately be of most use to practitioners and to theorists. As we study students* developing 
perceptions, we see that students did value open-mindedness and self-direction arid seemed to demonstrate it 
increasingly. We would like to be able to account for how students actually did undergo the changes that they 
demonstrated in their interviews. It is our conviction that communicating these subtle and complex patterns 
across disciplines to a wide range of faculty is a prelude to refining the criteria for assessment of these 
abilities. 

Finding: Outcomes are Abilities that Transfer to Performance at Work 

Alumnae Realized Career Expectations 

Alumnae surveys reveal that graduates were successful in achieving their immediate career-oriented goals. 
Over 90% of Alvemo alumnae were employed in fields related to their college major within six months of 
graduation. 

In one alumnae survey, we found that these alumnae were more likely to obtain professional positions than 
their mothers; education clearly seems to function for them as an effective route to professional careers. 
Graduating seniors had higher career expectations than alumnae were able to realize in two years, but alumnae 
rated aspects of satisfaction with their first positions and potential for advancement as above average. 
Alumnae also show a more positive attitude about their college learning after two years than seniors express at 
the time of their graduation, although both groups rated their college experience as above average on a 
majority of items (Mentkowski, et al., 1983). 

Given their generally positive attitudes toward college preparation, how did alumnae abilities carry over to 
performance? Faculty had identified intellectual and interpersonal abilities for the new curriculum based on 
experience as educators and professionals and on an analysis of academic disciplines and literature reviews. 
But would these same abilities form a basis for effective performance at work after college? 

Abilities Learned in College 
Are Used by Alumnae 

We found a remarkable congruence between the abilities graduates say they used in the work place and tho? 
that Alvemo educators consider important outcomes of college. Intellectual and inteipersonal abilities were 
both identified as necessary for coping successfully with a range of situations. Abilities in general functioned 
as an organizing principle for role performance and careering. 

Professionals' perceptions of abilities descriptive of outstanding performers were congruent with demonstrated 
abilities on the job for managers but not for nurses. Why this is the case is not clear, but it reinforced our 
decision to assess the effective work performance of alumnae in addition to their perceptions. The fact that 
alumnae have focused on developing abilities during college might make for more congruence, but this cannot 
be taken for granted. Since career variables like salar> and status were not linked to effective performance for 
women in an emerging field like management, colleges seeking to validate their curriculum for women 
alumnae need to rely on performance as their indicator in addition to measures of perceptions and self-report 
indicators of career advancement. 
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Competence is a Concept and an Ex\, ^rience 

College outcomes and work were related very strongly by the notion of self-perceived competence. It was a 
cognitive organizer for learning ooth in college and at work. It was one of the most powerful experiential 
triggers for development according to two-year alumnae constructions about how they manage their career 
changes and career satisfactions. 

In sum, much of the power of outcome-centered education comes from the fact that the mere act of identifying 
outcomes as abilities and giving people a chance to practice them has an impact that carries from college to 
later life. Often, educators attempting outcome-centered education are in large part responding to the press for 
work-usable education without sacrificing what education has traditionally meant. It seems possible to take a 
liberal education and define it in terms of ability outcomes that students can make part of their experience, 
creating an effective, lasting link between education and the world of work without having to sacrifice the 
value of those complex outcomes/abilities that inform the commitment of the liberal educator. The 
outcomes/abilities of liberal education can be identified, and when identified and experienced by the student, 
do persist. 

Therefore, we make the following recommendation for a national wsessment system. 



Recommendation #2. A national assessment system should 
define abilities as having multiple components, and as 
integrated, developmental and transferable, so they make 
sense to educators and employers. 



IMPLICATIONS, ISSUES, AND QUESTIONS 

• How are abilities best defined? 

How abilities are defined is a critical issue. If performance criteria are to function as standards, then definition 
of abilities will remain a matter of some debate, and will continue to focus research for some time to come. 
We have found that this is a continuing search; in fact, Alvemo faculty, with ongoing review and changes in 
ability definitions, just issued an official redefinition of four of the eight abilities in 1991. 

• What docs it mean to assess both cognitive and affective components simultaneously? 

There is been a good deal of discussion in the literature (Astin, 1991; Erwin, 1991; Tcrcnzini, 1989) about 
the importance of measuring cognitive and affective components of abilities in college outcomes studies. We 
have concluded that cognitive and affective components of abilities appear simultaneously in the performance 
of abilities, and therefore, ability components should be assessed in performances that engage a range of 
components simultaneously. That means cognitive and affective components should not be separated in 
assessments that are designed to measure impact of college learning on abilities demonstrated in work and 
citizenship. 

• How do we measure content and abilities, knowledge as well as performance? 

Messick (1982) and Wolf (1989) argue forcefully that it is not just the amount of knowledge that is at issue, 
but also the organization or structure of knowledge that acts as a functional system for performance, 
particularly for abilities like problem solving. 
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Jessup(1991) comments that the knowledge underpinning one element of an ability may also be relevant to 
many other elements. He argues that "when the body of knowledge is taught separately from the practice of a 
profession it tends to become an end in itself, developing its own structure and priorities, with the result that it 
does not necessarily relate closely to practice" (p. 126) The same holds true for assessment. Jessup discusses 
the dilemma of assessing knowledge in relation to performance in this quote: 

[Regarding]... the dilemma of assessing knowledge in relation to performance or as a body of knowledge 
largely separate from performance, the ideal solution is perhaps to do both. Where a coherent body of 
knowledge exists, and where it is also studied for reasons other than practicing a related profession, there is 
justification for pursuing it separately. It will also no doubt be assessed separately. But this should not be 
regarded as sufficient evidence or as a direct substitute for assessing the relevant knowledge in relation to 
professional performance. When theory is developed separate from professional practice it takes on a life of 
its own with its somewhat different values, priorities, constructs and language. It tends to pursue lines of 
development and address problems of intellectual interest, rather than the problems identified in the practice 
of the profession.. .Even when the theories are relevant, the connections with practice have often not been 
adequately established during training to facilitate their application... An analysis of the knowledge whrh 
people actually draw upon, and need to draw upon, to perform competently, may not appear in what is 
taught as the body of knowledge underpinning a profession or occupation... or, if it is covered, may not be 
accorded the priority it deserves. Competent professionals tend to acquire a set of guiding principles, of 
which they are often only partially conscious, derived largely from their experience. These may build upon 
'academic' theories and knowledge or be only loosely related. While this is recognized in areas such as 
management, it also appears to be true in well established professions such as medicine... it would seem 
necessary to...assess the knowledge required to underpin and extend competent performance, directly in 
relation to such performance... this should occur even if the knowledge is assessed separately as pan of an 
academic discipline (Jessup, 1991, pp. 126-127). 

• How define construct validity when abilities are multidimensional rather than unitar,- constructs? 

Furth',r, the definition of abilities as having multiple components counters measurement of critical thinking— or 
other broad ability identified in the National Goals— as a unitary construct. That is important because most of 
the available psychometric procedures treat constmcts as unitary rather than multidimensional (Frederiksen, 
1986). Further, when we are just beginning to define these abilities, our definitions will shift over time, as we 
lcam from the experience of trying to measure them. What is construct validity when abilities are not fully 
defined or when definitions of the abilities emerge in part during the assessment process, as an assessor is 
judging a unique performance? These are clearly issues that still need resolving (Mentkowsk\ 1989, 1990). 

• What is good evidence? 

Performance assessment is tugging at the edges of traditional measurement. Here, the unit of analysis, that is, 
"good evidenc e," is expanding from student selection of predetermined test item alternatives-or even short 
answers— to include proactive, open, interactive, dynamic, sustained student performance (Loacker, 1991). The 
measurement strategies described in later sections in this paper include such performance samples that are 
measured in context. This means that the abilities and accompanying performance criteria that define them 
cannot be assessed outside of a particular context, or it will not be clear whether the evidence is sufficient or 
not, nor what that evidence should be (Wolf, 1989). Wolf acknowledges the difficulty of making correct 
inferences about abilities without large amounts of evidence, and of acquiring adequate evidence by focusing 
only on the ability without the contextual referent. Often, one sees only some components of an ability. How 
then can we be sure we are seeing it? Clearly, multiple sampling in situations after college at work is 
essential. 
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• What is the developmental sequence of abilities? 

Perhaps an even more critical issue has to do with the developmental sequence of abilities. Alvemo faculty 
consistently assess for levels of abilities, defining them in terms of beginning, developing, and advanced 
abilities (Alvemo College Faculty, 1991; Diez, 1990; Roth & Riordan, 1984) (see Appendix A, F). First, our 
research suggests that abilities not only developed at different times, but that they were sequentially related in 
the world of work. Our study of outstanding managers and executives enabled development of a competence 
model which confirmed that some abilities arc sequentially related to others, and suggested that dispositional 
components of abilities like socio-emotional maturity undergirded development of more specific, job-related 
abilities like efficiency actions or proactivity. Our study of nurses showed that the helping abilities did not 
require the sophisticated knowledge ai. i onceptualizing skills that coaching abilities did. Coaching went 
beyond strategic influencing when nurses specified the direction of behavioral change toward increasing the 
responsibility of others. Clearly, a good deal of research will need to accompany the specification of abilities 
as developmental. 

• How measure changes in abilities after college? 

Further implications follow from the conclusion that effective assessment means linking information on how 
students come in to a program to how they graduate, and relating those changes to the curriculum (Astin, 
1991; Mentkowski & Doherty, 1983 revised 1984; Mentkowski & Strait, 1983). When an assessment system 
is committed to measuring change over time, strategies that assess complex, higher order abilities may 
generate changes in abilities that may be qualitative, nonadditive, and nonlinear. The appropriate model is 
development and change, not consistency across time, or continuation of a similar ability. 

Assuming change means that prediction will not necessarily rest on an additive model. That is, we will not 
necessarily expect to see "more of the same" after instruction or after college, but rather, qualitatively different 
learning outcomes. How do we show change? What kinds of measures will validly take this assumption into 
consideration (Collins & Horn, in press; Willett, 1988, 1989)? 

Further, if abilities are developmental, and they continue to be adapted by alumnae after college, as our 
research suggests, then this will have implications for the assessment of college-learned abilities in the 
workplace. Messick (1980) argues that assuming change rather than consistency should influence our 
expectations. Assuming change means that prediction will not necessarily rest on an additive model. "How 
measure development of abilities after college? To reiterate, the appropriate model is development and 
change, not consistency over time, or continuation of a similar ability. 

• How define reliability when we assume change rather than consistency? 

When we are looking for a complex ability in a student performance at graduation, will we expect to see more 
of the same if we look for that ability after one year in the workplace or five years after graduation? Or will 
we expect to see a qualitatively different ability, particularly when he or she demonstrates that ability in a 
setting that is quite different from college? When our developmental model assumes change, rather than 
persistence of an ability, it does not follow that persons who do not demonstrate the ability, cannot develop it, 
or even that they cannot demonstrate it. What kinds of measures will validly take this assumption into 
consideration? 

What happens when we add the assumption that development is, at times, nonlinear? Alverno longitudinal 
study results (Mentkowski, 1988) suggest that individuals recycle through earlier forms of thinking when they 
meet new situations. Here, change is not always consistent, "upward" movement on measures of development. 
How does this assumption shape the assumptions underlying models of assessment where one is demonstrating 
that college-learned abilities make a difference in how abilities are performed in work and citizenship roles 
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after college? What kinds of assessments are valid under these conditions? And how do we define reliability 
when change rather than stability is the rule, when we assume change rather than consistency (Mentkowski, 
1989, 1990)? In addition, what the components of abilities are, and how well they can be specified in 
performance criteria will continue to challenge us. 

• Can all outcomes be specified? 

i 

Probably not, but the effort to do so is an exciting one. As one sees students learning and alumnae 
performing as a result of understanding what to perform, one is motivated to continue the effort. The danger 
in the approach is that specification becomes an end in itself, and we begin to get so specific that behaviors 
are no longer linked to the nexus of motivation, attitudes, knowledge, self-perceptions and skills that underlie 
complex performances. 



Principle Learned #3. Thinking critically, communicating effectively and solving problems arc abilities 
common to college education and work. Effective performance at work is integrated; it L made up both 
of intellectual and interpersonal abilities. 



In this section, we make the case that to effectively make education-work relationships, the abilities of 
thinking critically, communicating effectively, and solving problems, identified in the National Goals, should 
become part of faculty and employer expectations. Research reported here suggests that effective performance 
at work was likely made up equally of interpersonal and intellectual abilities that are integrated in the 
performance. Interpersonal ability was an important outcome of college that is also essential to the workplace 
and should be incorporated with the threr, outcomes already defined. It is important to assess these abilities in 
order to demonstrate relationships between education and work. 

Examining effective performance at work includeu looking at adaptation to new settings. If individuals are 
somewhat at sea in new settings, what do they use as a lifeline (Mentkowski, 1988)? What kinds of abilities 
are needed to carry the individual through a variety of new settings and situations? Do these abilities become 
integrated— that is, does an individual actively construct them into a complex whole and show coordinated use 
of multiple abilities in a situation, such that the performance of each ability is supported by the others? 
Cognitive-developmental models of life-span growth (that is, moral, intellectual, and ego development), were 
used as college outcomes measures in our longitudinal study. These describe ways of thinking. They do not 
describe what people do in situations, their ways of performing. To examine integrated performance that 
enable on-the-job functioning, it was necessary to move beyond intellectual, ego, and moral mindsets to actual 
behavior. To do that, we researched broad, holistic abilities or competences, two terms we use 
interchangeably to describe learned dispositions io perform in ways that reflect specialized knowledge, 
attitudes, motivation, self-perceptions, and skills. 

Our research suggests that the broad, faculty-defined abilities that students develop across the Alverno 
curriculum do become integrated and do allow for role performance, What is interesting for understanding the 
development of integrated performance is the combining of these abilities. While intellectual development and 
competence in intellectual abilities have long been the cornerstone of general and professional education, our 
research indicates that interpersonal abilities are just as important. Indeed, integrated performance on the job 
demands both sets of abilities. To examine the evidence, we now turn to three data sets from students, 
alumnae, and professionals who are not Alvemo graduates. 
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Finding: Students Synthesized Intellectual and Interpersonal Abilities 

In looking for relationships among the cognitive-developmental patterns, learning styles, and generic abilities 
we measured, we found the following unexpected hut valuable result (Mentkowski and Strait, 1983): 
Students' performance on the battery of twelve measures, at entry to college, and again two years later, tended 
to cluster statistically around two separate developmental factors (one we call logical or analytical thought and 
one we call socio-emotional maturity or interpersonal ability), but after four years in college, the two clusters 
had merged. This may reflect one of the most desired outcomes of college—namely, that students integrated 
their own understanding and use of these two kinds of abilities. If this is the case, personal integration may 
assist in keeping them from being intellectually, emotionally, and morally "undone" in new unfamiliar settings. 

Let's explore this integration theme further. Are intellectual and interpersonal abilities integrated once the 
individual faces the world of work? Because the Alverno faculty had identified intellectual abilities and 
interpersonal ones as curricular outcomes (Alverno College Faculty, 1979 revised 1985), they wanted to know 
whether these abilities would lead to effective performance at work after college. Some interesting results 
emerged from studies of Alverno alumnae and from studies of practicing professionals who are not alumnae. 
Evidence from the two independent data sources shows that both intellectual and interpersonal abilities are 
critical for effective work performance. 

Finding: Alumnae Stressed the Importance of Both Intellectual and Interpersonal Abilities 

In analyzing alumnae interviews 3 , we found two major categories of complex abilities that were equally 
important in managing their work roles and careers. Both younger and older women, across all professional 
groups, cited reasoning abilities— using such terms as analysis, problem solving, decision making, planning, 
and organizational abilities— as important to their career performance. But alumnae also consistently 
emp^Tsized interpersonal abilities learned in college as critical to their effectiveness on the job (Giencke-Holl, 
ct al, 1985; Mentkowski, ct al. 1983). 

Some abilities identified as crucial to effective performance, such as reasoning abilities, on the face of it, were 
similar to those identified by most college and professional school educators. In contrast, interpersonal 
abilities, long an expected result from informal learning alone, were critical to effective performance as 
evidenced in the observations of working alumnae. The finding also suggests that all highly complex 
cognitive abilities should be integrated with high-level interactive ones. Clusters of abilities carried forward 
from college to the world of work. When integrated and adapted to the workplace, they contributed to 
effective performance. The assessment of highly complex cognitive abilities should be assessed through 
performances that also engage iuteractive and interpersonal abilities. 

Finding: Practicing Professionals Also Used Both Intellectual and Interpersonal Abilities 

The Alverno studies of effective managerial and nursing performance were designed to build a bridge to 
professionals who were not Alverno alumnae in order to validate abilities the faculty had identified (DeBack & 
Mentkowski, 1986; Mentkowski, et al, 1982) 4 . 



Of 60 Alverno seniors studied intensively near graduation, 32 participated two years later in three-hour semi-structured, confidential 
interviews of their perceptions of their abilities, learning, and personal growth. 

For the management study, local business leaden nominated 146 women managers and executives as effective. Over 100 of these from 
53 private corporations met our criteria for "manager" and provided us with job performance interviews and careering histories; 
perceptions were assessed through ratings of abilities critical to education, selection, and performance (Klemp, 1978; McClelland, 1976). 
The job performance interviews were analyzed for managerial competences (Boyatzis. 1982; Evans, 1982) that characterize effective on- 
the-job performance. For the nursing study, three health care settings (acute care, long-term care, and a community agency) were selected 
for their excellent reputation in both lay and professional communities so that more effective performance was more likely measured. 
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The fallowing is a list of the resulting managerial competences in order of the frequency with which they were 
actually performed in our study, from most often to least often: proactivity, diagnostic use of concepts, 
development cf others, accurate self-assessment, efficiency orientation, expressed concern with impact, 
conceptualization, self-presentation, perceptual objectivity, oral communication, use of unilateral power, self- 
control, management of groups, positive regard, use of socialized power, logical thought, stamina and 
adaptability, spontaneity, specialized knowledge, and concern with affiliation. 

Intellectual and interpersonal abilities had equal importance for these professionals. Managers were equally 
likely to use intellectual abilities (thinking through problems, applying past experience to interpret events, 
using a framework to guide analysis and actions) as interpersonal abilities (using power, developing 
subordinates, managing groups). 

The set of competences identified in the study of nurses consists of nine abilities. These are defined in a 
codebook of performance criteria. The following is a list of these nursing competences in order of the 
frequency with which they were actually performed in our study, from most often to least often: helping, 
influencing, independence, coaching, conceptualizing, ego strength, emotional stamina, reflective thinking, and 
positive expectations. The negatively coded competences, ranked for frequency of negatively coded incidents, 
are: conceptualizing, independence, positive expectations, ego strength, and emotional stamina. Influencing is 
an ability that calls for intellectual as well as interpersonal skills. It involves providing a rationale for a 



Eighty-three female staff and supervisory nurses were selected through peer nomination as outstanding performers and were interviewed 
along with a group of those not nominated. These nurses, from a wide range of units in the three settings, provided job performance 
interviews, careering histories and perceptions of abilities ethical to education, selection, and performance. 

The set of managerial competences consists of four ability clusters: Socio-emotional Maturity, Intellectual Abilities, Interpersonal Abilities, 
and Entrepreneurial Abilities. Each of these clusters is described by several competences, and each competence is elaborated through 
a set of behavioral descriptors. The four clusters, together with the competences, are presented below. These are defined via a codebook 
with performance criteria. 



Competence Model for Effective Managerial Performance 



Soclo-emotlonal Maturity Intellectual Abilities 

Self-control Logical thought 

Spontaneity Conceptualization 
Perceptual Objectivity Diagnostic use of concepts 



Specialized knowledge 



Interpersonal Abilities Use of unilateral power 

Development of others Use of socialized power 

Expressed concern with impact Concern with affiliation 

Accurate self-assessment Positive regard 

Stamina and adaptability Management of groups 



Self-presentation 



Entrepreneurial Abilities Oral communication 

Efficiency orientation 
Proactivity 



Competence Model for Effective Nursing Performance 



Conceptualizing 
Emotional Stamina 
Ego Strength 
Positive Expectations 
Independence 



Reflective Thinking 
Helping 
Influencing 
Coaching 
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desired behavior, including appealing to a higher motive and providing valid information in order to change an 
attitude or behavior. But it also means persuading by a variety of strategies, refocusing negative emotions to 
constructive issues, and persuading someone to follow the nurse's example. Effective nurses used coaching to 
change client attitudes and behavior, which means rewarding desirable behavior or giving positive feedback »n 
other ways, providing information to increase the other's responsibility, and encouraging others to take more 
responsibility for their own care. In conceptualizing, they created patterns of data, identified health problems, 
and gave rationales for treatment pla*ij. In demonstrating emotional stamina, they did not allow emotions to 
interfere with performance, but controlled anger, overcame fear, or responded calmly when attacked. 

Intellectual and interpersonal abilities shaped effective, integrated performance of effective nurses, managers, 
and alumnae. Therefore, we make the following recommendation. 



Recommendation #3. A national assessment system should 
assess for abilities already identified in the national goals 
and should provide for performances that integrate 
intellectual and interpersonal abilities. 



IMPLICATIONS, ISSUES, AND QUESTIONS 

That effective staff and supervisory nurses, as well as managers and executives showed integrated performance 
of these abilities in situations is consistent with the frequency of observing performance of several 
competences simultaneously in a single situation. To ensure effective career performance for their graduates, 
educators have to focus not only on the development of cognitive skills but also on their integration with high- 
level interpersonal ones. Multiple assessment strategies are needed to also tap both intellectual and 
interpersonal abilities. 

• What does it mean to assess different components of abilities simultaneously? 

Again, abilities that have multiple components and that are often integrated in a single critical incident or 
event in the workplace have intellectual and interpersonal components. Ability descriptions that emerge in 
studies of professionals cited in this paper consistently include both. If we follow the above recommendation, 
we can still expect a good deal of frustration both in specifying abilities and measuring them in context. 

• How assess integrated performance? 

Understanding what it means to assess for integrated performance is one of the more challenging issues in 
sampling graduates' work or service performance. In Recommendation 2, we argued that we need to assess 
cognitive and affective components simultaneously. Still another kind of integration is called for in the 
graduate bringing together specialized knowledge and his or her performance in a particular task or setting. 

Here we are led again to argue that effective performance at work is integrated; that it is made up of both 
intellectual and interpersonal abilities. By now we are confronting implications for measuring post-college 
abilities at work. Measures need to consider the performance that can enable us to observe integrated 
performance. So far, we have used both performance interviews that rely on behavioral events and 
perspectives interviews that measure perceptions of integrated performance, and the actual combining abilities 
in situations. Here we acknowledge the issue, "How assess for integrated performance?" We will find the 
question again in the next section, as we discuss another form of integration-between thought and action. 
This kind of integration is a component of self-sustained learning, which we will now discuss as a critical 
clement in transferring college-learned abilities to the world of work. 
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Principle Learned #4. To effe' My transfer college-learned abilities, students need to develop learning 
to leam skills, or self-sustainu* .earning. Assessment mat incorporates feedback and opportunities to self- 
assess fosters self-sustained learning. 



Self-sustained learning that links education to work is an important outcome of college that should be added to 
the three outcomes already defined in the national goals. Self-sustained learning is a key ingredient for 
enabling the college graduate to link education to work; i.e., abilities learned in college transfer to work 
because the student has learned how to learn. This ability enables continued learning, adaptation and 
integration of abilities, and further learning within a particular job situation and across situations and positions. 

Finding: Self-Sustained Learning That Links Knowing and Doing 
Is a Causal Element in Integrated Performance 

We have learned that (1) growth occurs during college as the result of a particular curriculum, (2) this growth 
includes increasingly sophisticated moral and intellectual mind sets as well as integrated performance of 
interpersonal and intellectual abilities in situations, and (3) individuals may nevertheless recycle through less 
sophisticated forms of thinking when confronted with new decisions and situations. Education seems to make 
a difference in development of personal growth and integrated performance for both Alverno students and 
alumnae, as well as practicing professionals in management and nursing. 

But what learning processes account for these outcomes? What kind of teaming yields integrated performance 
of intellectual and interpersonal abilities, particularly at work? 

To answer this question, we turn to longitudinal interviews of student perspectives. 3 The following 
description is adapted from Mentkowski (1988). 

Self-Sustained Learning in Students 

The interview analysis found that learning as described by students was a process of experiencing, reflecting, 
forming new concepts, and testing one's judgment and abilities in action. Two aspects of learning that 
students described seem to be characteristic of i traditional liberal arts education: (1) Students arc taught to be 
objective and to stand back and reflect on their experience; ana (2) They are consistently exposed to 
opportunities to form new concepts, complete readings, and attend lectures on theory. But these students also 
came to realize that hands-on experience is critical for learning. They also avowed that using new knowledge 
is necessary to really learning it and that one must test new-found skills. From the students' perspective, these 
elements fit together as a process that describes how they leam. That process undcrgirded the self-sustained 
learning necessary for effective performance at work. 



5 We gathered perception data through a confidential, open -ended interview ranging up to three hours in length, guided by a protocol 
of questions and probei (Much & Mentkowski, 1982, 1984). Because this measure is lengthy and complex, both to administer and 
to analyze, we selected samples for interviewing from both longitudinal and cross-sectional study populations. The results here 
reflect over 320 interviews from 82 students who provided interviews at one-year intervals at the end of each year in college, and 
again, five years after college. In all, 461 interviews were collected and transcribed from 129 participants. Interviews from 
traditional-age students and those from alumnae have been analyzed so far. 
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SELF-SUSTAINED LEARNING: STUDENTS 

Learning is: 
A process of experiencing, 
reflecting, 

forming new concepts, and 

testing one's judgment and abilities in action. 



One student, "Andrea," described this process: "By learning something and going and applying it, you can find 
out what worked for you, what didn't, what you really don't understand, and then you can go back and 
question. Or in learning new theory, you can apply, you can think back to the clients you have had or the 
situations you were in and say, 'If I had known that then, it could have helped me a lot.' So I think the 
application and then going back to theory and questioning helps make it more solid in your mind— you can 
understand it better." 

Changes in Student Learning Styles 

Thus, our student interviews independently confirmed Kolb's (1984) recent description of the learning process 
as experiencing, reflecting, forming new concepts, and testing one's judgment and abilities in action. 6 Further 
evidence from Alverno longitudinal studies of sn dent changes in personal growth are the dramatic changes 
appearing in students' orientation to learning styles, using Kolb's Learning Style Inventory (1976). At 
entrance, both younger and older students showed marked preference for concrete experiencing over abstract 
conceptualizing and for reflective observing as against active experimenting. In the first two years, they 
moved rapidly toward a more balanced pattern: By the second testing, they had come to rely equally on 
concrete and abstract modes and to show a similar flexibility in choosing either a reflective or an active 
approach. Additional analyses revealed that students who showed high achievement in the curriculum changed 
more and that when age, pretest scores, and other variables were controlled, the curriculum still accounted for 
change. Overall, achievement in the curriculum had a greater effect than age on changes in learning style 
preferences. 

These results provide another clue that certain kinds of learning may be related to integrated performance. 
They also suggest that experiential learning is an important curricular clement (Doherty, Mentkowski, and 
Conrad, 1978; Mentkowski, O'Brien, Cleve, and Wutzdorff, 1983). But which elements of the curriculum 
matter? 

Curricular Elements Most Important to Self-Sustained Learning 

To what curricular elements do students attribute "learning to learn"? The interview analysis identified three 
major components that describe the development of this process of learning to learn: taking responsibility for 
learning, making relationships among abilities and their use, and using different ways of learning. Figure 3 
illustrates the learning outcomes and the curricular elements to which students attribute the development of 
their leaming-to-learn skills. 




' Kolb derived thii experiential learning theory from theories by Lewin, Dewey, and Piagct and has researched it in pan through measures 
of learning style. 
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One student, "Andrea," described this process: "By learning something and going and applying it, you can fin' 
out what worked for you, what didn't, what you really don't understand, and then you can go back and 
question. Or in learning new theory, you can apply, you can think back to the clients you have had or the 
situations you were in and say, 'If I had known that then, it could have helped me a lot.' So I think the 
application and then going back to theory and questioning helps make it more solid in your mind-you can 
understand it better." 

Changes in Student Learning Styles 

Thus, our student interviews independently confirmed Kolb's (1984) recent description of the learning process 
as experiencing, reflecting, forming new concepts, and testing one's judgment and abilities in action. 6 Further 
evidence from Alvemo longitudinal studies of student changes in personal growth are the dramatic changes 
appearing in students' orientation to learning styles, using Kolb's Learning Style Inventory (1976). At 
entrance, both younger and older students showed marked preference for concrete experiencing over abstract 
conceptualizing and for reflective observing as against active experimenting. In the first two years, they 
moved rapidly toward a more balanced pattern: By the second testing, they had come to rely equally on 
concrete and abstract modes and to show a similar flexibility in choosing either a reflective or an active 
approach. Additional analyses revealed that students who showed high achievement in the curriculum changed 
more and that when age, pretest scores, and other variables were controlled, the curriculum still accounted for 
change. Overall, achievement in the curriculum had a greater effect than age on changes in learning style 
preferences. 

These results provide another clue that certain kinds of learning may be related to integrated performance. 
They also suggest that experiential learning is an important curricular element (Doherty, Mentkowski, and 
Conrad, 1978; Mentkowski, O'Brien, Clcve, and Wutzdorff, 1983). But which elements of the curriculum 
matter? 

Curricular Elements Most Important to Self-Sustained Learning 

To what curricular elements do students attribute "learning to learn"? The interview analysis identified three 
major components that describe the development of this process of learning to learn: taking responsibility for 
learning, making relationships among abilities and their use, and using different ways of learning. Figure 3 
illustrates the learning outcomes and the curricular elements to which students attribute the development of 
their learning-to-leam skills. 
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Figure 3 - Student Learning Outcomes and Their Causes as Attributed by Students 



STUDENT ATTRIBUTED CAUSE 



STUDENT OUTCOME 



Instructor attention and empathy, 
feedback, Self-assessment 



-> 



Taking responsibility for 
learning 



Experiential validation, 
Instructor coaching, 
Professional application, 
integration of abilities 



-> 



Making relationships among 
abilities and their use 



Practice, Feedback, 
Modeling, Peer learning 



-> 



Using different ways of 
learning 



Note that one of the more prominent causes gleaned from the interview examples is experiential validation, 
having to apply abilities within and across courses, demonstrating them on assessments and during internships, 
and using the abilities in multiple settings. This may be a curricular element essential for integrated 
performance. Further, feedback and opportunities to self-assess experienced in Alvcrno's ability-based 
performance assessment process, led to taking responsibility for learning, for learning to learn and assess on 
one's own. 



While students are showing development of self-sustained learning, how do we know that students value 
learning? Does learning become part of their value system? And do students' professional values include the 
kind of learning that would support integrated performance? 

Student interview data confirm that students' commitment to personal, career, and professional values 
developed throughout college. There is also an important performance dimension to conmitment: students 
consistently gave examples of how they were acting out of their personal and professional value systems. 
Like their understanding of learning, their understanding of their own commitment developed throughout 
college, and they show more and more sophisticated behaviors that match their increasingly internalized goals 
and values. 

Students also consistently broadened the settings in which they described themselves using their abilities. As 
they progressed, they cited instances from work, family, and other environments as often as their in-class 
assessments. This is a sign that they also have cognitively made the transfer that they claim to have made 
cxpcrientially. Through experiential validation of their abilities, students were able to construct a justification 
for liberal learning in which personal growth and effectiveness mediated between educational experience and 
concepts of professional role performance. Thus, there is reason to believe these students' valuing of liberal 
learning contributed to their capability for integrated performance. 



Though students integrated learning performance, do two-year alumnae continue to show self-sustained 
learning after college at work? Do they value learning on the job? While Argyris and Schon (1974) have 
described the importance of a theory of action in effective performance and McClelland (1975) has shown that 
abilities are a basis for effective role performance, researchers and educators still must demonstrate how this 
happens. Educators want to know how and why abilities are developed so they can foster them in college. As 
we analyzed our 32 alumna interviews, a picture of active, self-sustained learning began to emerge as it was 



Coming to Value Liberal Learning 



Self-Sustained Learning in Alumnae 
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practiced at work. Several components of learning to learn characterized alumna behavior: (1) Learning is a 
continuous process. Alumnae regarded learning opportunities as motivating career development and job 
choice; (2) Learning to learn for alumnae means to tie knowledge, theory, and experience to productive action 
and to put these elements together in new situations; (3) Alumnae apply abilities in action, get a response, and 
adjust performance or ideas accordingly; (4) Alumnae show integration and adaptation of abilities based on 
the experience of observing, thinking, retrenching; and (5) Central to self-sustained learning is thinking and 
performing in particular settings and situations. 



SELF-SUSTAINED LEARNING: ALUMNAE 

Experience learning as a continuous process 

Tie knowledge, theory and experience to 
productive action 

Apply abilities in action, get a response, and 
adjust accordingly 

Integrate and adapt abilities 

Think and perform in context 



Following are preliminary descriptions of what we have categorized as developmental, acceptance, and 
credentials orientations to learning, based on interviews with five-year alumnae, using the Alumnae 
Perspectives Interview (Mentkowski & Much, 1980, revised 1985; Deemcr & Mentkowski, in progress). 
These are value constructions that seem to underlie the way alumnae orient themselves to continued 
development and adaptation of their abilities at work. 

We are currently analyzing interviews for learning, confidence, self-assessment, reflection, and adaptation 
outcomes. So far, there seem to be differences evident in the way some individuals think and feel about 
learning. 

A key difference in the orientations revolves around what individuals hope to achieve through learning and 
what they value. A person demonstrating a developmental orientation is intrinsically invested in a 
performance and its outcomes. In contrast, a person demonstrating an acceptance orientation is invested more 
in the secondary outcome of social acceptance. The credentials orientation is also tied to secondary outcomes 
derived from the performance, in this case the potential for career mobility, salary and status. 

Needs addressed through performance arc clearly not orthogonal and the outcomes are not viewed as mutually 
exclusive. Most, if not all, individuals were concerned about tangible achievements; they valued their social 
relationships. These issues were not absent in the discourse of individuals with a developmental orientation. 
However, the weight or centrality of these issues seemed to vary; it influenced how individuals construct 
learning situations. 

The orientations depict what seem to be an underlying motivator for the individual but may be domain specific 
in its expression. For example, persons may be developmentally oriented in their civic involvements but 
engage in paid employment for financial reasons. For this reason the centrality of the domain(s) of action for 
such persons may be a necessary preliminary discrimination in uncovering a developmental orientation. 

3c, 
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Similarly, the acceptance orientation may be contingent on context for its expression, i.e., caring about what 
particular people think about her performance. Although the orientations are played out in specific 
contexts— an observation that impacts our ability to see evidence of the orientation-they are thought to reflect 
a central source of drive and a central way of constructing meaning that influences their continuing 
construction of learning situations. 

In the developmental orientation, learning is intrinsically valued and enjoyed. > Opportunities to learn and 
experience new situations and challenges are viewed positively as promoting self development. The learner 
regulates her involvements where possible, setting realistic goals within the framework of her values and 
interests. She assesses her performance in terms of the outcomes of the performance itself and its impact on 
the situation. The learner seeks constructive feedback to improve her performance. Her self worth is not 
contingent on her performance. Embedded in this orientation is a sense of personal agency emanating from 
the individual. She seeks out opportunities, is futuristic in her thinking, acts proactive, and demonstrates an 
internal locus of control. She shows confidence, self-respect, and ability to express herself, but also sees 
learning as a collaborative venture, and personal relationships may provide additional motivation. She shows 
an ability to make relationships and to engage in independent learning. 

In the acceptance orientation, learning is oriented toward significant others in the situation, and their 
expectations. Meeting these expectations and receiving approval from others is critical to feelings of self 
worth. The learner is oriented toward concerns about looking incompetent, and losing others* respect as a 
result. This learner thrives on recognition and feelings that others are proud of her. Self doubts, unrealistic 
goals, and working to please, with less regard for personal cost, may result in manifestations of stress. 

Her orientation is to just "getting work done" done when approval is not at stake. The feedback of others is 
given more weight than the learner's own assessment of performance, and may be seen as a threat to 
validation by others, or as a means to figuring out how to achieve recognition. This learner may be motivated 
by threats and intimidation as well as more positive expressions of pride. Also embedded in this orientation is 
an inability to make relationships between the performances expected of her and how her performance serves 
her own needs; she may sacrifice her own development to give others what she perceives they want. 

In the credentials orientation learning is motivated by ideas of advancement. It is an instrumental sacrifice 
maintained through discipline. Thus learning activities are cast as something one has to do, not what one 
wants to do. This learner uses feedback to determine whether her performance will be acceptable and what 
she will need to do to improve. She completes learning activities through self-discipline, driven by anticipated 
career, status, or financial rewards. 

While we do not yet know how these value orientations to learning characterize our alumnae as a whole, our 
current longitudinal interviews, from entrance to college to five-year alumnae, will enable us to determine 
what value frameworks alumnae use at work, particularly in relation to self-sustained learning (Dcemer & 
Mentkowski, 1990). 

In sum, college learning and abilities form a foundation for role performance after college, but learning to 
learn is a prerequisite to adapting abilities in a given role one has. Self-sustained learning is a process that 
enables adaptability to multiple settings; it enables students to perform abilities learned in college, later at 
work (see Figure 4). Thus, this study of alumna and student interviews suggests another kind of integration, 
between knowledge and its use. 
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Linking Knowing and Doing for Integrated Performance 

The relationship between knowing and doing intersects the central issue of whether integrated performance 
will continue to accrue for the individual as the result of participation in a setting other than college. From 
our initial study of student and alumna interviews, it seems that integration not only of intellectual and 
interpersonal abilities but also of thought and action in a particular context is key to a sustained pattern of 
growth and change. If we center our educational strategies so that they consistently ensure relationships 
between knowing and doing, we are more likely to effect an integration between thought and action that will 
generalize to a variety of situations, including those that call for integrated functioning within a framework of 
values. 

In a later section, we examine whether individuals taught in this way continue to act responsibly in 
organizations. There are enough examples in the interviews *e are analyzing in our current five-year follow- 
up study to indicate alumnae are doing so. The link between thought and action, between knowledge and 
behavior, seems to develop during college and develop further afterward. More important, it seems to be a 
cereal building block in the glimpses of integrated performance we have seen so far. 

Our study of managers and executives also confirmed the importance of knowledge linked to action. We 
researched abilities through performance interviews and a perceptions inventory. We also asked participants 
for their perceptions of what outstanding professionals should do. Managers were invited to judge a range of 
performance characteristics. We found that effective managers generally performed abilities they 
independently judged as characteristic of outstanding performers. The findings also allowed us to identify 
those abilities that professionals did not demonstrate but identified as important— such as negotiating and 
networking in management-thus signaling abilities that should be part of the manager's repertoire. Other 
abilities, such as demonrraung self-control and using socialized power, wer more important for effective 
performance than the managers realize. However, the fact that we found this congruity between perceptions of 
managerial abilities and actual performance is evidence that the managers in this study had another element 
critical to the development of integrated performance— and thus, to sustained learning— i.e. knowledge 
reflected in action. 

Thus, application of abilities cannot be left to chance in professional school education. Traditional curricula 
have emphasized development of knowledge over action, just as they have emphasized intelectual abilities and 
ignored interpersonal ones. A curriculum should stress using abilities across multiple settings, and this kind of 
performance can be assessed (Loacker, 1991). A curriculum should also incorporate the fact that increasingly 
complex performance develops in concert with gradually more sophisticated modes cf thought. Results from 
all three data sources— students, alumnae, and professionals-confirm that the integrated performance of 
abilities is a developmental, learned process that needs systematic teaching and opportunity for practice, but 
can be assessed in the workplace through in-depth interviews of graduates' perspectives. 
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Recommendation #4. A national assessment system should 
assess self-sustained learning and should include feedback 
and opportunities to self-assess. 

IMPLICATIONS, ISSUES, AND QUESTIONS 

One implication that follows this recommendation is that different kinds of measures will be needed to 
adequately assess self-sustained learning. We discerned self-sustained learning both from perspectives 
interviews, which are about the way individuals construct concepts, orientations, and values about learning, 
and from performance interviews, where individuals recall events when they were demonstrating their abilities. 

• How assess self-sustained learning? Is it feasible? 

Our experience suggests that performance interviews (e.g. Behavioral Event Interviews described later) are 
effective in measuring some components of self-sustained learning, such as the ability to self-assess, but that 
other kinds of indepth interviews of how the individual constructs learning are needed to adequately measure 
transfer of self-sustained learning beyond college. Thus, it is a combination of modes and methods that leads 
us to a clearer picture of self-sustained learning. 

Is the measurement of self-sustained learning feasible? Currently, we are using computer programs to assist 
with the content analysis of interview text; but until these approaches are more well-developed, sampling is 
recommended as a concession to feasibility issues. However, the performance interviews do get at some 
components of self-sustained learning, including self-assessment. Since self-assessment is such an important 
element that fosters self-su stained learning, some of the more complex element are dealt with, using the less 
feasible perspectives interviews. 

The U.S. Department of Labor (1991) report, "What Work Requires of Schools/* also notes the importance of 
measuring Knowing How to Learn (uses efficient learning techniques to acquire and apply new knowledge and 
skills). This ability is considered an important part of the foundation for performing effectively at work. 
Understanding how important it is to include this ability will certainly keep sustaining us in our search for 
more feasible measures. We have worked to develop a Student as Learner Inventory that asks students to judge 
how characteristic various elements of self-sustained learning are in their approaches to college learning. (This is 
discussed in Loacker's (1991) position paper.) This is a way to identify changes in student perspectives during 
college that also generates feedback to students. It gives faculty a way to intervene in learning to learn skills 
early in a student's college career. 

• How assess for integrated performance? 

We return to this issue, which emerged in the last section. Again we find the need to measure multiple 
components of abilities in situations. Here we find still another elaboration of integrated performance: linking 
knowledge to action that generalizes to a variety of situations. We recognize the complexity of this question. In 
Learned Principles 5 and 6, we turn to more developed descriptions of our work in assessing post-college 
performance, in an effort to lay out methods that deal with this issue. 
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• How give feedback and opportunities to self assess? 

The systematic measurement of all of the components of self-sustained learning is generally less feasible than the 
measurement of other complex abilities. Our experience with indepth perspectives interviews and performance 
interviews suggests that eliciting full descriptions of self-sustained learning requires patient, interactive 
interviewing. Nonetheless, there are pedagogical reasons for conducting reflective indepth interviews of 
participants' perspectives. By providing a reflective opportunity for the participant, they can address some of the 
concerns about how a national assessment system could efficiently give effective feedback. At the end of an 
interview, the interviewer would be in a position to give some feedback on abilities that were described. 
Interviewers could be trained to assist participants to self-assess. At the end of an interview, an interviewee 
would be in the position to think broadly about how abilities are related to the interviewee's current goals. Thus, 
the "feedback 11 occurs at the time of the interview, so there is no need to go to the expense of creating or 
delivering individual feedback to the interviewee at some future time. Our students and alumnae have often 
commented that these performance and perspectives interviews have led them to self-insights, because just 
participating in the interview gave them a chance to self-assess. When we ask participants to generate abilities 
that they feel are important, we have found that they are able to do this and that this helps bring the interview to 
effective closure. Using a reflective (perspectives) interview in a national assessment system would broaden this 
process, enabling participants to generate their own learning from their contribution to a national assessment 
system. 



Principle Learned #5. Comparing faculty-defined abilities to those demonstrated by outstanding 
professionals enables faculty to identify abilities students need for particular professions. 



Along with Alvcrno College, educators and researchers in a wide range of professions (business, industry, 
government, military, and health professions) have sampled work performance as an efficient tool for curriculum 
development. This approach represents an important expansion of method, for educators developing curricula 
and identifying abilities have often relied extensively on their broad expertise or on their supervision of students 
who are encountering professional settings. In reviewing content, faculty typically use literature reviews to 
extend their perspective on professional development. Such reviews tend to include a knowledge foundation of 
the profession, and may address technical entry level position requirements, but may not be sufficient to prepare 
the individual to function effectively across professional settings or across increasingly complex professional 
positions (McClelland, et al., 1990, Appendix D). 

On the other hand, performance interview accounts enable curriculum developers to: (1) identify abilities with 
enough contextual information to support educational interventions (Flanagan, 1954; Klemp & McClelland, 
1986), (2) link curriculum development to the abilities that are most critically associated with continued 
effectiveness in the profession (Boyatzis, 1982; DeBack & Mentkowski, 1986; Klemp, 1988; Mentkowski, et al, 
1982), and (3) develop communicable examples of professional and liberal arts abilities and values that can 
transfer across context (Stark & Lowther, 1988). 

Usually, educators sample work performance through some kind of interview that asks the person to recall an 
incident or event. The goal is the identification of abilities or competences that make for effective or outstanding 
performance. Direct observation is usually deemed both inefficient and too costly for several reasons. First, the 
object is not a task analysis that contributes to job descriptions. One can observe someone working over a long 
period of time without discerning an event or incident that would call up essential abilities, or abilities that are 
critical for effective performance. Further, direct observation often fails to sample the context for a behavior 
adequately. (For example, is heroic resuscitation life-saving or counter to a living will?) 
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Flanagan's (1954) Critical Incident Technique (CIT) and McClelland's (1978) Behavioral Event Interview (BEI) 
have been used to identify abilities for the professions. Although CIT is widely known for its effective 
application in establishing performance requirements for certification and licensure (e.g. Hubbard, Levit, 
Schumacher, & Schnabel, 1965), examples of the application of CIT and BEI to curriculum development, are not 
as widely available (Wilson-Pessano, 1988). We are not aware of any institution except Alvetno that uses the 
method for post-college studies of alumnae. 

Nevertheless, researchers from diverse institutions and professions have used these two related methods in 
long-term curriculum-embedded studies designed for improvement. They collected events that stood out to 
interviewees as examples of more or less effective performances. The CIT yields events that are relatively 
bounded in scope in order to support the sorting of events into unique categories of performance (cf. Fivars & 
Gosnell, 1966), v hile the BEI yields events with more elaborated descriptions of the thoughts and feelings of 
the interviewee at the time of the event in order to support the coding of performances for contcxtually 
demonstrated abilities (Boyatzis, 1982; Mentkowski, et al, 1982; Rogers U Reisetter, 1989). Analysis of the 
BEI may be supported by either an interviewer's summary write-up based on notes and review of the taped 
interview (Finch, Gregson, & Faulkner, 1991) or else by a write-up that organizes the participant's transcribed 
description into an event sequence (Rogers & Talbott, 1990). While there are a number of applications of this 
method in the United Kingdom (e.g., Bradshaw, in press); two examples in the United States illustrate the 
application of these methods. 

The Institute for Health Care Research in the American Institutes for Research (Wilson-Pessano, 1988) used 
the Critical Incident Technique to collect descriptions of effective and ineffective parental asthma management 
behaviors to establish objectives for an asthma management education program. Analysis of 574 reports from 
physicians, other health professionals, and parents yielded 130 distinct ineffective and effective asthma 
management behaviors, which were organized in five major areas of responsibility— symptom intervention, 
symptom prevention, use of medical and educational resources, communication among caregivers, and child 
development and family relationships. Analysis also revealed the knowledge, attitudes, skills, and specific 
behaviors needed by parents to carry out these responsibilities. 

Priorities for the curriculum were established based on the relative importance of the identified behaviors to 
control of symptoms, safely of the child, and overall well-being of the child and family, as well as the 
likelihood that parents would have difficulty carrying cut the responsibilities without educational assistance. 
Careful consideration was given to the choice of techniques to develop each parental ability, including specific 
behavioral change techniques (eg. self-monitoring, performance feedback, behavioral contracts), "job aides," 
and videotape components to address specific didactic problems or as triggers to discussion of emotional and 
attitudinal issues. Nurse-educators were involved, with educational specialists, in all stages of the development 
of the program. By focusing on the instructional objectives to be met, pretesting the educator's "script" and 
teaching activities, and assessing the effects on parental behavior, the educators practiced and thereby 
developed a much broader repertoire of skills in patient education than they previously possessed. The 
resultant program is currently being evaluated in a randomized control clinical trial to determine its medical 
and other effects on children and families. 

The Virginia Tech Office of the National Center for Research in Vocational Education (Finch, et al., 1991) 
used the Behavioral Event Interview to establish curriculum content for leadership development as part of a 
four-year project Project goals were to identify leadership attributes for vocational education administrators 
and then to develop resources for preparing successful administrators. Analysis of 272 events from 39 
vocational administrators and 78 instructors across seven states was conducted concurrently with data 
collection in order to facilitate both data collection and analysis (cf. Lofland, 1971; Miles & Huberman, 1984). 
Thirteen types of events were identified and an existing leadership framework and a set of leader attributes 
were verified and elaborated (Finch, et al., 1991). 
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After reviewing the needs of the profession and existing training resources, two resources were developed. 
One was a set of case studies that provide user., with opportunities to explore alternate actions and solutions to 
problems (the case studies are organized into the 13 functional groups; the content is drawn from interview- 
based write-ups that include supporting contextual information). Second, a simulation was created that 
provides participants with opportunities to utilize and assess their leadership capabilities as they assume 
administrative roles. The attempt to create a simulation also led the developers to adapt the BEI to the 
simulation. Vocational administrators were also asked to provide crucial events that demanded their full 
attention. The CIT was then used to efficiently gather additional vignettes in order to supplement stimulus 
materials for the simulation. 

As mentioned earlier, the Office of Research and Evaluation at Alvcmo College has also conducted a number 
of studies using the Behavioral Event Interview to support the development and validation of an ability-based 
curriculum that includes performance assessment. 

Finding: Faculty-Defined Abilities Relate to the World of Work 

Results from studies of outstanding professionals in nursing and management were described earlier. The 
following study, using similar methods, involved Alvemo faculty directly in interviewing and analyzing the 
information to find out how the abilities they defined relate to the workplace (Schall, et al., 1984). Eighteen 
faculty and staff from 13 disciplinary and professional areas conducted 134 performance interviews and also used 
elements of the Job Competence Assessment Process. 7 (See Appendix E.) Seventy-three professionals at the 
managerial level across 66 Milwaukee area organizations completed the interview and a Performance 
Characteristics Inventory (McBer, 1984). Organizations represented the following work environments: 
Agricultural Products, Business and Other Services, Communications, Education, Entertainment, Finance, 
Industrial High Technology, Manufacturing, Public Utilities, Real Estate, Transportation, Wholesale, and 
Retail. 

In the study, faculty and staff identified abilities professionals used and characteristics they perceived as 
necessary for their work, and related them to abilities taught in the Alvemo curriculum. Professionals 
interviewed spoke from a specific job context (i.e., a particular job performed within a given career field and 
type of organization), thus enabling faculty and staff to establish direct relationships between abilities taught 
and those performe d at work. Results provided a catalog of abilities thai were identified by area professionals 
as critical to success in a variety of career paths, were demonstrated on the job, and were related to abilities 
taught at Alvemo. 

A second finding was that type of organization influenced the kinds of abilities coded from the interviews. 
For example, a greater number of professionals from high technology environments reported developing others 
and showing positive regard for employees, while professionals from manufacturing and agricultural products 



7 Job Competence Assessment Process. In the Alvemo studies, we adapted Job Competence Assessment, a mote extensive technology 
that includes the Behavioral Event Interview. As noted earlier, research has effectively linked this assessment technology to curriculum 
development in college and professional schools. McBer and Company (Appendix E) (McClelland; 1976; McClelland, et al., 1990; 
Boyatzis, 1982;) and Cambria Consulting (Klemp, 1982) have effectively used the methodology to create training in various 
organizations and also to create a performance appraisal system that includes the Behavioral Event Interview. In fact, performance 
appraisal systems are redesigned based on the criteria and abilities identified through initial Job Competence Assessment studies of 
outstanding performance in the workplace. The Job Competence Assessment Process (Appendix E) is designed to identify competences 
that are not only related to effective performance, but that cau*? effective performance. Competences are characteristics of persons who 
are effective, although McBer recognizes that individual competence must be considered within a system that includes the person, the 
job, and the work environment within which the two interact Basically, the method involves defining performance effectiveness 
criteria, and choosing a criterion sample (e.g. peer nomination, customer ratings) to distinguish outstanding and good performers. These 
individuals are interviewed with the Behavioral Event Interview (McClelland, 1978), which is coded for competences in order to create 
a competency model of effective abilities. In a Performance Characteristics Inventory, these participants rate a series of abilities and 
skills as relevant, critical for outstanding performance, and essential for selection and training of new employees. 
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industries seemed to rely on demonstrating a concern for impact and proactivity. While the high technology 
manager was interested in bringing people along within the organization, and believed that people can 
improve, the manufacturing and agricultural manager focused more on the image or reputation of self or 
business, and madf; being a resourceful initiator a top priority. This people orientation on the part of high 
technology managers was interpreted as reflecting a more advanced orientation to "Total Quality 
Management," and supported faculty efforts to define abilities in ways that reflect future as well as current 
position responsibilities. 

The most important finding from this study may have been that the researchers could relate faculty-defined 
abilities to all categories of abilities coded in the interviews: socio-emotional maturity, entrepreneurial 
abilities, intellectual abilities, and interpersonal abilities. This was some verification that broad abilities which 
Alverno graduates must demonstrate were similar to those that professionals report using and perceive as 
necessary for their work. This argues again that complex abilities are integrated and performed in context, and 
that a single situation may elicit a number of abilities simultaneously. 

Finding: Five- Year Alumnae Performances Relate to the World of Work 

Five-year alumnae in managements nursing, and other professional areas described performances in these areas 
as well as in a personal activity (e.g., child rearing, graduate learning, volunteering). A generic taxonomy 
(Appendix C) of 29 alumnae abilities was developed, based on a synthesis of previous studies and curriculum- 
defined abilities (see Appendices A, B) and the coders 9 attempts to capture what stood out in the performance 
examples. 

Alverno researchers also developed codes for dynamic settings (eg., system level failure, values conflict, 
development of others, interpersonal conflict) in order to relate the broad context of the performance to the 
curriculum. Case examples were organized around abilities associated with most effective alumna 
performance, as well as by four of these dynamic settings that are common across professions. Write-ups of 
these now provide students with case examples of performance that they can discuss in relation to both liberal 
arts and professional abilities. Transferability of curriculum abilities to new settings is demonstrated to 
students through the juxtaposition of alumna performances that have similar dynamic settings, but different 
levels of effectiveness and different professional settings. 

This experience suggests the following recommendation for a national assessment system. 



Recommendation #5. The method that some educational 
institutions and consulting firms use to measure abilities in 
the workplace to improve student and employee performance 
can be used as a strategy in a national assessment system. 



IMPLICATIONS, ISSUES, AND QUESTIONS 

The recommendation suggests that the performance interview method can be used to sample abilities after 
college, in context, as well as for identifying abilities. 

There are several implications for designing assessment. First, researchers and educators agree that the 
performance interview technology should be used to assess performance in context and assess those abilities 
that a person uses to guide performance across a range of situations (e.g., "theory of action/ 1 Argyris & Schon, 
1974). Further, the interview provides the link between abilities and behaviors that are critical elements in 
identifying developmental criteria for assessing performance in context; that is, in situations where we expect 
the person assessed to create or construct a response that may differ for every individual. Because these 
abilities will be causally related to effective performance, faculty and employers can use the information 
generated both for improvement and accountability purposes. 
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• Can the method assess abilities in college and at work? 

The Behavioral Event Interview method has been adapted successfully in college as a systematic way to assess 
abilities related to success in college and life. In the Student Potential Program developed by the Council for 
Adult and Experiential Learning (CAEL) with the assistance of McBer and Company and with funding from 
the Ford Foundation, 15 institutions participated in a study of the method as a way to assess behavior patterns 
not measured by standardized paper and pencil tests (Council for Adult and Experiential Learning, 1986). An 
evaluation conducted by the Higher Education Research Institute (Astin, Inouye & Korn, 1986) found that the 
Student Potential Program interview (the Behavioral Event Interview approach adapted for CAEL use) has a 
significant degree of construct validity, and adds significantly to the multiple prediction of traditional 
admissions measures, suggesting that the Student Potential Program (SPP) is measuring relevant academic 
abilities not measured by traditional criteria. 

Alverno participated in the CAEL study. As one outcome, the education faculty currently use the measure as 
a strategy for measuring the Education Department's abilities (Diez, 1990; cf. Appendix B) as an outcome 
measure after a student's first and second field experiences, and as an entrance assessment for the student's 
third and fourth field experiences. The method has also been developed as an in-class assessment 
accompanying student internships (Mcntkowski, et al., 1983). 

This is some evidence that the measure can assess abilities in college, supporting its use both in college and at 
work. This argues that perfomiance samples could be collected using the same strategy. Further, the studies 
cited earlier using the method in the workplace confirm the feasibility of use in the workplace. 

• How sample faculty/institutions to build the research base for creating abilities? 

All of the studies cited in this section suggest these kind of studies could be "sampled" to build a broad set of 
abilities. Clearly, there are questions about the degree to which any set of abilities and the attendant 
developmental performance criteria would adequately represent those that institutions would claim as outcomes 
for their graduates. But there are enough collaborative efforts that suggest that such an effort is possible at 
both the disciplinary and professional level (Association of American Colleges, 1991; Loacker, 1991). On our 
own campus, we are currently coordinating a Kellogg grant involving 1 1 institutions that span high school 
through medical school curricula; the institutions share abilities to investigate whether we can build a 
sequential set of abilities that reflect levels of education. 

• How feasible is it to follow graduates after college? 

Literature sources such as ERIC and Social SciSearch show few if any citations regarding formal study of 
post-college employment experiences or performance in relation to curriculum. Beyond graduate follow-up on 
surveys, institutions may use a variety of methods to monitor their graduates' post-college experiences as a 
means of program improvement. Often this is informal in nature and is seldom published even in the form of 
technical reports. The potential flow of information back into programs occurs naturally and informally 
whenever there are practical fieldwork or other clinical experiences and an institution's graduates are entering 
the supervising agencies (e.g. schools, hospitals, businesses) that provide placements. Professional schools in 
both undergraduate and graduate schools, because of their responsibilities to a well-defined market and to the 
fulfillment of a public trust, are more likely to seek and receive information about their graduates informally 
and through systematic studies. (Locally, at least four colleges, Alvemo, Cardinal Stritch, University of 
Wisconsin-Milwaukee and University of Wisconsin- Whitewater carry out mentoring and first-year teacher 
support programs that indirectly serve teacher preparation curricula.) 
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• How feasible is it to assess for graduates 1 performance at work? 

Admittedly, post-graduate assessment of actual performance in the workplace as a measure of college 
outcomes presents many challenges. As Galluzzo and Craig (1990) point out in relation to teacher education, 
a wealth of relevant information about preparation can be collected as teachers enter practice, but, they predict, 
many institutions will be reluctant to use such data if it is used only for accountability purposes. In practice, 
many individual and contextual factors can confound the interpretation of traiiung impact. Data to be used for 
improvement often needs to provide additional information than that used for traditional accountability 
frameworks (e.g., accreditation); and institutional supports that integrate evaluation data into program 
improvement are seldom more than cursory. 

The research of Adams and his colleagues completed in the early 1980's, stands out for its use of longitudinal 
follow-up studies. Aycrs (1986) also produced some singular work involving longitudinal obseivations o 
developing teachers, in practice, as an evaluation model. Galluzzo and Craig (1990), however, point to several 
factors that limit more effective use of evaluation measures, including resistance to measuring performance 
once graduates leave the institution. As meaningful as such efforts may appear from the perspectives of 
curriculum development and educational policy, little use seems to be made of them because of their lack of 
sensitivity to individual and contextual factors that influence performance at work. Institutional commitment 
to substantive evaluation structures is also an issue. 

In medical education, the precedents for assessment of performance following education have been stronger. 
Indeed, medical schools seem to take a much more active role in following up graduates than some of the 
other professions (Miriam Friedman, personal communication, September 1991). Relationships between 
medical school performance and post-graduate competence have been investigated both generally (i.e., to 
predict success in clinical practice) and for specific program evaluation (Gonnella & Hozat, 1983; Gunzburger, 
Frazier, Yang, Rainey, & Wronski, 1987; Veloski, Herman, Gonnella, Zeleznik, & Kellow, 1979; Woodward, 
1984; Woodward & Ferrier, 1982; Yindra, 1988). Different forms of performance assessment have also been 
explored as a basis for targeting training support to improve practical skills (e.g., Norman, Tug well, & 
Feightner, 1982; Norman, Neufeld, Walsh, Woodward, & McConbcy, 1985; Finkel & Norman, 1973). 

• What are barriers to using information about graduate's performance at work back on campus? 

What are the barriers to using back on campus, information from assessment at work? A potential question 
posed by college and university faculty and administrators is: "To what degree will we be held accountable 
for graduates' performance in the work place, when that performance may be more influenced by the context 
in which the graduate is working than by the education that the graduate received in college?" How long can 
faculty be expected to be held responsible for graduates' performance? One year? Five years? How does one 
sort out the effects of context in the performance of graduates, so that performance can be fairly attributed to a 
curriculum the graduate may have experienced earlier? 

More recently, Dinham and Evans (1991) make the case that measures such as accreditation standards are far 
too limited to provide adequate evaluation back to schools. Measures that move beyond the curriculum— such 
as post-college performance-are necessary to provide adequate support to 
institutional improvement. 

Thus, two important elements of post-college studies are important to establish. The first is creating a context 
for post-college assessment that makes public a shift from assessment as information for accrediting or 
demonstrating accountability, to also using assessment as information for improving curricula. 
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Second, post-college studies should provide information on the definition of abilities as well as of performance 
criteria. Currently, they provide information for the "What abilities should we be teaching?' 1 question more 
than the question "What criteria enable us to judge these abilities in performance and distinguish effective 
performance?' 1 Performance data on graduates should link the conceptualization of the abilities and 
assessment. The resulting refined abilities will reinforce connections between instruction and assessment. 

• How can performance at work be linked to college instruction? 

Another issue has to do with the relationship between assessed performance and instruction. How can 
performance in the workplace be linked to instruction in college? Unless the information from post-college 
studies relates directly to instruction, some faculty reason, how can it really help improve one's teaching? 

• How assess for pcrformance-in-context? 

What is critical? How do we assess performance-in-context after college in a way that facilitates the analysis 
of what should be taught in college? Once faculty are teaching for abilities that arc expected to transfer, part 
of their interest is in the performance of those abilities in context and in the opportunity to assess it to see 
whether students have internalized the abilities and can perform them in a range of situations. This is one of 
the arguments for the traditional emphasis in professional education on student teaching, nursing clinicals, and 
opportunities for practice in on- or off-campus "laboratories." Faculty are only too aware that abilities are not 
likely to transfer unless the student can perform in context. 

Thus, assessment of students in context is already built in to most professional school curricula. Once an 
improvement agenda is welded to post-college assessment of graduates, faculty and administrators are likely to 
rely on performance-in-context as the kind of evidence that is critical for determining whether graduates are 
actually transferring college learned abilities to work. 

It would be unfortunate if post-college assessment of graduates would be used only for decisions on 
accreditation of professional programs. Administrators or faculty will hardly want to open themselves up to 
accountability demands that stem from lack of performance in situatioas in which thcy-or their 
graduates-have little if any control. 

Again, one interesting observation from our studies of alumnae is that when graduates found themselves in 
situations where they could not perform their abilities— that is, where the kind of work situation was not 
conducive either to demonstrating their competence or to further learning— they began to look for other 
positions, and changed jobs. This finding argues that it is critical for college students to develop independent 
self-sustained learning that provides them with the independence and skills to exit work situations that they 
cannot change. It also suggests that assessments of graduates after college should incorporate ways to define 
and judge the context in which they are working, for the degree to which the context calls for or undercuts the 
performance of the abilities being measured. 



Principle Learned #6. Complex abilities that connect education and work— including self-sustained 
learning-can be assessed in graduates' work performance in a variety of professional contexts. Some 
abilities can be linked to college learning, and some distinguish effective performance at work. 



How and to what degree does the college graduate contribute to effective performance at work? Pascarella 
and Terenzini (1991) have summarized the research on how college affects student outcomes during and after 
college. Considerably less research has addi :d the long-term effects of college than has addressed the 
first-order question of what impact college has on the student while still at college. Although the evidence is 
spotty in some domains, Pascarella and Terenzini concluded that college results in long term gains in learning 
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and cognitive development, attitudes and values, psychosocial changes, moral development, and socioeconomic 
outcomes, as well as in gains in quality of life. Surprisingly little research has directly addressed the question 
of how on-the-job performance of the college graduate makes a contribution to conceptualizing the work to be 
done or moving projects to effective completion. Does level of education distinguish effective performance? 

Finding: Level of Education Distinguishes Professionals' Effective Performance at Work 

In our studies of professionals in uie work place, we found that some abilities were developed mote through 
education than experience on the job. Nurses with a baccalaureate degree were more likely to act 
independently. They took responsibility for their judgment, or took an advocacy role for another, while at the 
same time, they showed ego strength by acting responsibly even at the risk of incurring disapproval. The 
more educated nurses not only demonstrated more influencing (they made more strategic attempts to refocus 
emotions or to change behavior), but they also exhibited more, coaching which is a more complex behavior 
because it goes beyond strategic influence when a nurse specifies the direction of behavioral change toward 
increasing the responsibility of others. The more educated nurses were better at conceptualizing, and showed 
more reflective thinking. They tended to search for or find greater insights by reflecting on their own 
behavior, feelings, and beliefs and their consequences. The less educated nurses were more likely to 
demonstratively fail at conceptualizing by not recognizing relevant relationships or by failing to guide their 
actions according to relevant higher order principles. They made conceptualizing errors that put clients at risk. 
Thus, this kind of conceptualizing should be pan of any kind of assessment of nursing graduates' performance 
on the job because it discriminates effective performance. 

While level of education was less related to performance for women managers in our study, those who had 
completed a management training program showed more stamina and adaptability and were more likely to 
work with others in dealing with subordinates, rather than using their authority via administrative fiat. Those 
who showed more rapid advancement in their company and most likely a broader range of experience, 
demonstrated more accurate self-assessment and better-developed self-presentation skills. A more 
serendipitous finding also emerged. A small group of women executives interviewed communicated a holistic 
approach to life— as well as to management-that they want to teach and share. We did not set out to measure 
this, but it struck us that these executives were sure of their commitments and consistently acted out of them. 
These executives demonstrated integrated performance and involvement and tended to shape their own 
organizations. 

What are the implications for assessment? Given when this study was conducted, many of these successes of 
these early women executives may be ungeneralizable, because their performances were honed by the 
experience of making it in management at a time when they were not quite so welcome in the field. Can 
management graduates today count on a well-defined and sustained education that contributes to integrated 
performance and to their personal growth? Will such an education really make a difference later, or will only 
a few develop the potential to contribute to the kind of corporations and organizations that we need in the 
twenty-first century? 

Are there other implications for assessment? Clearly, performance interviews should be accompanied with 
questionnaires that tap a number of indicators, such as level of education, years of experience, and type and 
level of position. After all, level of education, for example, did distinguish effective performance at work in 
studies of graduates from a range of professional programs. These studies argue that assessment of graduates' 
performance at work should distinguish further among those abilities that should be developed in college from 
those that are better developed on the job. If faculty and employers are to collaborate in education and 
training, respectively, it will help to know which abilities should be taught where, to the degree that it is 
helpful to make those distinctions. Further, if abilities are defined as developmental, then it makes sense that 
some abilities should be measured during college and at graduation, whereas we would not expect other 
abilities to surface until the person has had some degree of work experience. 
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The crucial question may not be how the performance of the college graduate differs in the aggregate from the 
non-graduate, but how do we characterize the most effective performances of college graduates? When 
college educators envision their students transferring their college learning to post-college performances, what 
kinds of post-college performances by their graduates do they expect will make an effective difference? 

How do we assess effective performance of college graduates? The formulation of this question has several 
premises. (1) Faculty have a significant stake in defining what they will be teaching for. (2) Faculty invested 
in making a difference in the work force or in effective citizenship will need to invest in understanding post- 
college performances and performance settings of their alumnae. (3) Summaries and examples of most 
effective performances by graduates can help faculty make the link between abilities learned in school and 
abilities performed on the job. (4) Faculty can make links between what students are learning in their study of 
the disciplines in college and the kinds of performances that students make. 

Our study of five-year alumnae (Mentkowski, et al., 1991; Rogers & Talbott, in press) demonstrates how the 
Alvemo Alumnae Performance Effectiveness Measure is beginning to give us a picture of how college faculty 
envision effective post-college performance. The Alverno Alumnae Performance Effectiveness Measure 
involves several evidentiary steps (see Figure 5) and has much in common with Job Competence Assessment. 
Both use the Behavioral Event Interview (McClelland, 1978; Mentkowski, et al., 1982; Rogers & Reisetter, 
1989) to obtain an indepth description of performance in context Both have procedures for developing write- 
ups that clarify the sequence of actions (McClelland, 1978; Rogers and Talbott, 1990) or code from transcripts 
(Appendix G). The method of determining abilities that lead to effective performance arc different, however. 
Job competence studies have helped generate codes related to effective performance by analyzing the 
difference between the performances of professionals who have been nominated as outstanding performers 
versus those who are not. Our alumnae effectiveness study has generated a comprehensive list of codes 
(Appendix C provides a summary of the codebook's abilities) based upon all of these prior studies and has 
supplemented these with ability definitions from the curriculum (Appendix B; Rogers and Talbott, 1991). 
Instead of nominating outstanding alumnae, we coded alumna performances using this comprehensive list of 
job competence abilities, and then had faculty from a range of disciplinary and professional areas 
independently sort these performances into levels of effectiveness. We were then able to generate a list of 
abilities most associated with effective alumnae performance by correlating their coding of abilities with 
faculty judgments of overall performance effectiveness (Rogers and Talbott, in press). (See Figure 5.) 

The faculty's evaluation of post-college performances is not congruent with the stereotype of an ivory-tower 
academic. Instead, the demonstration of achievement actions is the strongest correlate of Alverno faculty's 
judgments of effectiveness in post-college performances. Most generally effectiveness is associated with 
making a positive difference in an organizational context, including developing of others. A relationship to 
college education seems especially plausible for those alumna performance contributions that are based on 
conceptualization, diagnostic information-seeking, accurate self-assessment, specialized knowledge, and 
positive regard for others. 

The descriptions of alumna performance generated by the Behavioral Event Interview help faculty 
conceptualize links between education and work and provide them with examples of post-college performance 
that they can use to improve instruction and developmental performance criteria. 

Finding: Abilities Can Be Defined That Connect Education and Work 

In order to elaborate on our first recommendation, that a national assessment system should assess for abilities 
that connect education and work, we developed a schematic that sets forth the three abilities specified in 
National Goal # 5: Problem Solving, Critical Thinking and Communication plus Interpersonal Abilities and 
Self-Sustained Learning (Recommendations 3 and 4). We then categorized the abilities from the codebook 
(Appendix Q we have seen demonstrated at work and in service roles by Alverno alumnae (Figure 6). Those 
abilities associated with most effective performance of Alverno alumnae are also identified. 
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Figure 5 

ALVERNO ALUMNAE PERFORMANCE EFFECTIVENESS MEASURE: DESCRIBING EFFECTIVE POST-COLLEGE PERFORMANCE 



ELICIT ORAL DESCRIPTION 
OF PERFORMANCE: 
BEHAVIORAL EVEKT INTERVIEW 



What led up to situtation? 
What did you do? 

What were you thinking and feeling? 
Who was involved? 
What was the outcome? 



McC: Hand (1978) 
Mentkowski, O'Brien 
McEachern, and Fowler (1982) 

Rogers and Reisetter (1989) 



GENERATE "PERFORMANCE" 
WRITE-UPS OR TRANSCRIPTS 
OF INTERVIEW 



ll 



McClelland (1978) 

Mentkowski, O'Brien, 

McEachern and Fowler (1982) 
Rogers and Talbott (1990) 



RESEARCHERS CODE ABILITIES 
DEMONSTRATED IN PERFORMANCE 



Examples of Levels of Criteria: 

1. Conceptualization: Identifying and seeing 
key relationships between key issues 
while understanding the big picture. 

2. Participant generates ideas that meet 
one of the following broad criteria: 

• Ideas are disparate in time 

• Ideas donl seem to fit together until 
the concept is brought forward 

• There is distracting or irrelevant 
information 

• Ideas are systematically coordinated 

3. Participant ideas meet specific criteria: 

• Integrates patterns into coherent 
systems: develops system framework 
that takes into account available 
information 

• Sees connections between seemingly 
unrelated data, problems events, or 
situations 

• Differentiates an issue, problem, or 
concept into a set of constituent parts, 
citing a new concept as the basis for the 
differentiation 

• Integrates new component into existing 
framework, while maintaining a 
coordinate relationship among all 
components of the framework 

• States assumption or limitations of own 
framework or more than one framework 

• Uses unusual analogies to understand 
or explain the essence of an event 



FACULTY JUDGE GLOBAL 
EFFECTIVENESS OF PERFORMANCE 



1 



Adapted From Rogers and Talbott (in press). 
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ABILITIES ASSOCIATED WITH MOST 
EFFECTIVE ALUMNAE PERFORMANCE 



Achievement Orientation 
Achievement Actions 
Diagnostic Information Seeking 
Use of Socialized/Political Power 
Development of Others 
Conceptualization 
Accurate Self-Assessment 
Positive Regard 
Diagnostic Pattern Recognition 
Specialized Knowledge 



Figure 6 . 

Alumnae Performance Effectiveness Measure: Measured Abilities in Relation to National Goals 



PROBLEM SOLVING 



CRITICAL THINKING 



COMMUNICATION 



SELF-SUSTAINED LEARNING 



INTERPERSONAL ABILITY 



Proactivtty 

Achievement 
Orientation* 

Achievement 
Actions* 

Diagnostic Information 
Seeking* 

Diagnostic Pattern 
Recognition* 

""—ling/Systematic Thinking 

Lists Alternatives 

Reflectively Coordinated 
Practice 



Conceptualization* 



Formal Communication 



Hypothetical/Causal Thinking Use of Informational/Expert 

Influence 

Perceptual Objectivity 
Specialized Knowledge* 



Accurate Self-Assessment* 
Reflective Thinking/Valuing 
Active Development of Self 



Positive Regard* 

Sensitivity to Individual 
Differences 

Accurate Empathy 

Development of Others' 

Expressed Concern 
With Impact 

Development of 
Organizational Power 

Use of Socialized/Political 
Power* 

Use of Unilateral Power 



* Ability is associated with most effective performance of Alverno alumnae. 
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Note: Ability categories in the coding system, not related to the National Goals: Self-Control/Emotional Stamina, Ego Strength, Stamina, 
Spontaneity/Curiosity, Concern With Affiliation, Affiliative Action, Developing or Using Relational Power. 
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We are now going to take these relationships a step further, building off Figure 1 in "What Have We Learned 
#1." Now we will attempt to show how we can move from one ability, critical thinking, to specification of 
developmental performance criteria. 

Finding: Developmental Performance Criteria Can Be Linked to Aspects of Performance 
that are Most Critically Associated with Continued Effectiveness in the Profession 

How abilities are manifested through performance criteria does generally shift from college to post-college 
performance. For example, Alverno faculty expect that interdisciplinary, general education courses in the 
humanities will lead students to demonstrate £ range of behaviors characteristic of a critical thinker, and that 
these behaviors can be seen as developmental performance criteria. Faculty experience in teaching for and 
assessing for critical thinking has led to specifying these criteria as developmental. For example, students will 
first demonstrate systematic thinking, then critical self-consciousness of the thinking process, and ultimately 
reaching beyond the conceptual frameworks that currently shape one's consciousness (Appendix F). The goal 
of understanding the effect of assumptions and frameworks on critical thinking is also re-emphasized within 
the disciplines. Humanities faculty expect students to encounter a multiplicity of cultural assumptions and see 
how these guide their perceptions (Riordan, 1986; Roth, 1986). But, only some of the humanities general 
education criteria for a Critical Self-Consciousness of the Thinking Process (see Appendix F) transfer directly 
to performance after college, and those criteria that do transfer after college do not require, at this point in 
time, demonstration of an understanding of how thinking is culturally conditioned, (One may expect 
requirements to change as graduates become more involved in global concerns.) 

That criteria for college performance are related to but not always equivalent to post-college performance is 
demonstrated in Figure 7, "Some relations between developmental performance criteria for assessing critical 
thinking in humanities (general education) and criteria for assessing five-year alumnae." It seems that there are 
multiple disciplinary paths to the world of performance after college. For example, Alverno researchers who 
code alumna performance see perceptual objectivity (the demonstration of an understanding of more than one 
perspective) as an important aspect of managerial performance. Students can learn this broad ability through a 
humanities curriculum focusing on cultural assumptions; they can learn it through a psychology curriculum 
focusing on the multiple interpretations of behavior; or they can learn it through the consistent juxtaposition of 
different disciplinary frameworks in college. How students leam the ability in college will generally affect 
how faculty assess them for it, but how the ability is manifested after college can be assessed more generically 
if we focus on what is broadly transferred to post-college performance. By studying abilities of graduates 
after college, we can better identify what we expect to transfer from college to post-college performance. 
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ALVERNO COLLEGE 

Milwaukee, Wisconsin 



Figure 7. Some Relations Between Developmental Performance Criteria for Assessing Critical Thinking In Humanities (General 
Education) and Criteria for As? ;ssing Five Year Alumnae 



ASSESSED WCOU^HUMA ItfTIOD OtLOPieiTAl 
PERFORMANCE CRTEHA 1 



ABILITY COMPONBITS USEO M 
90TN COLLEGE AND WORK 



ASSESSED AFTER COUEGE PERFORMANCE CflTTERM SNiLAR TO 
HUMANITIES CflnBIA 



Systematic Thinking 

1, Strives to notice » much as possble in tho object of her thought (eg, a 
statement, a book, a workoUrt. behavior). 

2. Recognizes that then art appropriate sate o( concepts (specialized 
vocabulary) lor describing dflerent kinds of phenomena and general/ uses 
that language appropriately in communicating her observations. 

5. Distinguishes between what she observes and what she says about the 
object ol thai observation. ~ 

4. independently identifies and uses conceptual frameworks to organize and 
symhesize her understanding of that whch she has observed and to 
communicate that understanding to others. 

Critical Set-Consciousness of the Thinking Process 

4. Evaluates the perceptions judgments, and behavior of others relative to 
their context (i.e., withholds absolute judgment). 

5. Recognizos that experience may have muHpte layers of meaning; sees 
beyond the Meral meaning; shows tolerance for ambiguity, uncertainty, 
loose ends." 

S. Is wiing to question the assumptions and validity ol tor own perceptions 
and knowledge and those of others. 

7. Seeks authority (the knowledge and experience ol others) as a guide to 
understanding and action, but ultimately takes personal responsibly for all 
ol her judgments. 



Reaching Beyond the Conceptual Frameworks 
Which Currently Shape Joe s Consciousness 

1 . Actively explores complexity instead of ignoring or stepping around H 

3. Actively seeks a critical understanding ol alternate points ol view and ways 
ol knowing. 

4 Has the capacity to mediate between different points of view and to 
integrate (averse views 

5. Makes personal judgments only after examining multiple contexts and 
mufctye perspectives, and, even then, recognizes the limits ol her 
judgment. 



Systematic Thinking 

Strives to notice as much as possWe In the object of her thought (e.g., a statement, 
behavior). 

Recognizes that there are appropriate sets ol concepts (specialized vocabulary) for 
deserving different kinds of phenomena and goneraJy uses that language 
appropriately in communicating her observations. 

Distinguishes between what she obrwes and what she says about the object of that 
observation. 

Independently identities and uses conceptual frameworks to organize and synthesize 
her understanding of that which she has observed and to communicate that 
understanding to others. 

Critical SeHlonsclousness of the Thinking Process 

Evaluates the perceptions, judgments, and behavior of others relative to their context 
(it, withholds absolute judgment). 

Recognizes that experience may have multiple layers ol moaning; sees beyorv 
bteral meaning; shows tolerance for ambiguity, uncertainty, 'loose ends.' 

Is wibno to question the assumptions and vaidity of her own perceptions and 
knowledge and those ol others. 

Seeks authority (the knowledge and experience ol others) as a guide to understanding 
and action, but ultimately takes personal responsbitity for aH of ner judgments. 



Reaching Beyond the Conceptual Frameworks 
Which Currently Shape One s Consciousness 

Actively explores complexity instead of ignoring or stepping around it. 

Actively seeks a critical understanding of alternate points ol view 

Has the capacity to mediate between different points ol view and to integrate diverse 
views. 



Similar To Systematic Thinking 

Concepiualization; Identifies the most important issue(s) in a complex 
situation. 

Diagnostic Information Seeking: Uses questions to identify the specifics ol 
a problem or other situation. 

Diagnostic Pattern Recognition: Identities the discrepancies between the 
specifics ol a particular case and the more general pattern. 

Diagnostic Pattern Recognition: Distinguishes relevant from irrelevant 
information. 

Similar To Critical Self •Consciousness of the Thinking Process 

Perceptual Objectivity: Tolerating >verse Opinions: accepts or actively 
consioers differing opinions of others without premature foreclosure. 

Positive Regard: Expresses respect for others who are perceived as 
different 

Developing Others: Invites subordinates/colleagues to discus problems 
affecting performance 

Accurate Sen -Assessment (Commitment to Improve): Admits i weakness, 
mistake, or lack ol knowta toe, while recognizing the importance e' 
overcoming the process trou led to K 

Reflective Thinking/Valuing: identities and reflects on behavior, feelings, or 
beliefs and their consequences, ft may include reflecting on a weakness 
or mistake. This must result in showing or searching for new insight about 
sell. 

Proactrvdy: Takes responsbitity for own judgment and acts independently 
within a social frame. 

Ego Strength: Fulfills respcnsbikty at risk of incurring disapproval ol 
others. 



Reflective Thinking/Valuing: Relates decisions or actions to own values. 

Simitar To Reaching Beyond the Conceptual Frameworks 
Which Currently Shape One's Consciousness 



Diagnostic Information Seeking: Seeks information from mufiple sources 
to clarity a situation. 

Diagnostic Information Seeking: Seeks and gets information that clarifies a 
situation. 

Perceptual Objectivity: communicates to interviewer that she has an 
understanding ol the different perspectrves that each party brings to a 
conlict or interpersonal event 

Use ol Sociafeed/PoilcaJ Power: Uses awareness of own role and role of 
others in group in shaping the task and deciding about the process 



1 Adapted from Roth, R , & Riordan, T. (1984). Bohavjors characteristic of the critical thinker. Milwaukee, Wl: Arvemo Productions 
FR?r ) \ ' Ad^ted from Rogers, G, A Tabott, M (1991). Generic aUrty cc^es for scoring the Behavioral Event Interview ol Alverno Alumnae Mifwau.ee, Wl: Alverno Productions 
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The depth of learning in the disciplines is an important framework for learning to think that must be preserved 
in our formulation of what is transferred to post-college performance. Generic criteria do not directly address 
this depth of disciplinary learning. After the undergraduate experience, however, we can look for 
conceptualization that does have depth. In making comparisons of depth of thinking after college to thinking 
in college, it is important to recognize that thinking after college may be more difficult to do. Disciplinary 
study is a sustained and collective inquiry that colleges and universities have a special responsibility for 
promoting, but after college graduates will not have the systematic thinking of others laid out as an ongoing 
forum of inquiry into a specific topic that they are invited to join. Admittedly, depth of thinking is still a 
hallmark of good graduates* performance, as Figure 5 demonstrates. Here, in keeping with our focus on 
critical thinking for this example, we have listed levels of criteria for conceptualization in Figure 5, in order to 
reinforce the depth of thinking that characterizes post-college performance. This is an ability we code for in 
graduates* performance across both work and service roles. 

For Alvemo alumnae, conceptualization has been demonstrated outside of disciplinary-based inquiry. In our 
alumnae generic ability codebook, the various manifestations of conceptualization are described as identifying 
and seeing key relationships between key issues, while understanding the big picture. For our five-year 
alumnae, seeing key relationships is a contribution to work performance. 

Effective Alumnae Demonstrate Complex, Integrated Performance 

The kind of performance that integrates many situational aspects and levels of contexts can be contrasted with 
other performances that tend to be more limited or short term. In the Behavioral Event Interview, some of the 
most frequent abilities demonstrated in context involved very short bursts of ad hoc perfo»mance. For 
example, alumnae often initiated new actions, communications, proposals, or meetings to accomplish a task 
and often listed some alternatives or consequences (showing hypothetical if-then thinking). Although many 
alumnae contributed through these short bursts of performances, this kind of performance generally did not 
make a profound difference in the situation nor was it judged as particularly effective by faculty, even though 
it is contextually appropriate. While complex, integrated performances were less frequent (for example, 
making a conceptual contribution that integrates diverse issues or that connects individual initiatives to the 
collective organizational goals), thece complex and integrated performances stand out as making substantial 
contributions. They are the kinds of performances that colleges can probably take most of u A e credit for 
achieving. 

Given the evidence, we recommend the following: 

z. 

Recommendation #6. A national assessment system should 
sample graduates 1 performance at work and in citizenship 
roles, as well as in college, and judge performance in 
relation to those developmental criteria that distinguish 
effective performance. 



IMPLICATIONS, ISSUES, AND QUESTIONS 

There are two implications to draw here. While we have discussed performance interviews, we do not set this 
measure out as the only alternative. Rather, we have set forth a technique that we have confidence in and that 
is beginning to meet our goals both for improving curricula (defining abilities and building performance 
criteria) and for accountability (distinguishing effective performance). Other measures need to be developed. 

Institutions, employers, and alumnae need to work together to design and develop appropriate strategies for 
sampling performance through work samples, portfolios, behavioral event interviews, critical incident 
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techniques, or other assessment technologies ^Uready in place at work. Further, those involved in this effort 
will need to work further to identify abilities involved in "effective' 1 work and citizenship roles. We have 
provided some evidence that this information can be collected directly from college graduates at work and in 
service. Still, a number of issues remain. 

• How assess for integrated performance? 

This issues asserts itself again. While a codebook effectively enables us to identify abilities and their 
components, we will need to demonstrate reliable coding in order to ensure that we identify which abilities are 
integrated and see how they are combined in performance. Further, we know from perspectives interview 
data, that some alumna constructions that illuminate value orientations toward learning and self-sustained 
learning do not necessarily appear full-blown in performance interviews. There is a good deal of work to be 
done to define what we mean by integrated performance and how to recognize it when it occurs. 

• How assess performance-in-context? 

This question has not been dealt with fully, and our earlier discussions elaborate on the difficulties of the 
issue. Gearly, the degree to which abilities should be defined broadly or specifically will continue to 
challenge us, particularly since performance criteria used to demonstrate effectiveness of employees in the 
work place usually demand the i performance be related to effectiveness in a particular position. This raises 
still another question. 

• How account for differences in work settings and in professional roles? 

Clearly, it is difficult to account for differences in work settings and in professional roles. When assessment 
technologies are used in the workplace to identify abilities for training and selection, designers can generally 
rely on job descriptions or other role definitions that a particular organization has developed. Inched, one of 
the reasons the Job Competence Assessment Process was developed was because organizations generally did 
not know what abilities distinguished various roles. 

When moving to using the method in post-college follow-up studies, one is faced with developing ability 
definitions and developmental performance criteria that cross professional roles and contexts. We have dealt 
with this issue by coding for type of dynamic setting (e.g., system level failure, values conflict, development 
of others, interpersonal conflict), and by coding perfoimance in relation to the outcome ("What happened as a 
result"), within an extensive description of situational elements ("Who was involved? What led up to it?"). 

In our research, we continue to confront the issue of how variability in setting and roles affects our judgments. 
For example, we asked faculty representing a range of interdisciplinary general education areas to judge the 
relative effectiveness of examples of perfoimance from across a wide range of settings (graduate learning, 
child rearing, volunteering). We asked professional faculty to judge the relative effectiveness of examples of 
performances from within their profession, but their judgments still had to span different roles and settings 
within the profession. To evaluate how effectively we are dealing with this issue we will still have to pursue 
this in greater depth. 

• How use expert judgment to code abilities in performance? 

We have developed a number of sirategies to code abilities in performance, but we will not elaborate them 
here (Mentkowski, et al, 1982; Ropers & Talbott, in press). These strategies address selecting of assessors, 
describing the judgment process, analyzing use of criteria in relation to student responses, establishing inter- 
judge agreement, evaluating assessor training, analyzing assessor judgment and feedback, and analyzing 
performance in relation to criterion measures. 
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The effort to generate valid expert judgment and to describe and design processes for ensuring validity means 
investment. So far, Alvemo has been investing in the development of strategies for using expert judgment to 
code abilities in performance in the curriculum for 18 years, and in the Office of Research and Evaluation for 
IS years. We suggest that this investment is comparable to the value of the results. Faculty and external 
audience response to the data so far has outweighed the costs, or we would not continue to do use this 
technology. 

• How validate expert judgment of complex, integrated performance? 

One might argue that for deciding what modes of response are good evidence and determining the validity of 
the expert judge's interpretation of the response, we are relying on similar meas : "ement principles that we 
have used earlier. Therefore, all we are changing, basically, is that we are using qualitative rather than 
quantitative methods and procedures. In fact, -xploring qualitative methods, and applying the same validity 
criteria we have always applied is only a first step. 

Generally speaking, when we establish the validity of expert judgment, we ask for inter-judge agreement. We 
look to see if the judgment of one assessor is similar to that of another. And we have various statistical 
procedures to determine agreement or reliability. However, in situations where we are assessing complex 
abilities and integrated performance in complex situations, we might argue that we have to be careful that 
agreement between the judges does not get too high. Here is where quantitative measurement assumption 
breaks down. Generally, the higher the level of agreement, the better. But if we are asking two judges, one 
from Humanities and one from Behavioral Sciences, to judge performance of a student's performance, we 
know that we cannot specify the criteria for judgment entirely, that each judge will call forth the perspective 
of their discipline to make a global judgment. 

If the level of assessor agreement is too high, we might be asking for simplistic judgment of complex, 
integrated performance. Suppose a hospital nurse supervisor, a nursing faculty member, and the hospital's 
cthicist are making judgments on the ethical decision-making ability of a nurse as she resolves a complex 
ethical dilemma. Such a dilemma may involve, for example, a patient who is asking about whether he is 
dying, and she must decide whether to informatively respond that he is indeed dying, which would be against 
the wishes of the family and the doctor. The nurse supervisor, the nursing faculty member, and the ethicist 
may bring quite different philosophical perspectives to bear. Indeed, we would wish that to be the case. The 
alumna should be prepared to perhaps deal with the weight given to a more idealistic perspective by a nursing 
instructor whose responsibility is to develop independent, complex decision-making in nursing students so that 
the profession advances. The hospital nurse, on the other hand, may be weighing much more the practical 
judgment that considers the role of a nurse in a hospital and the limitations of her role. In such cases, inter- 
judge agreement, if it gets too high, may be cause for concern. Currently, our measurement theory does not 
account for such a situation. In short, reliability becomes a secondary consideration, and the methods that 
define reliability are limited in this application. 

We have a number of well-established strategies for validating expert judgment, which we describe in 
numerous Alverno publications (e.g., Alvemo College Office of Research and Evaluation/Assessment 
Committee, in press). We still have to sort through a number of issues related to establishing the validity of 
judgment when complex, integrated performance is elicited, and when assessors arc deliberately drawn from 
different perspectives and roles. 

• How judge effectiveness? 

Judging effectiveness is a complex task. Many of the common strategies open to us to get confirmation from 
external sources (employers, experts in the profession) present difficulties. In studies that cross such a wide 
range of situations and roles as alumna studies do, finding a common measurement is clearly difficult Nor do 
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some of the standard indicators such as level of position and salary serve these purposes. Our study of women 
managers found that position was not related to performance. Salary was often not a good indicator when 
geographical pay scales and private and public sector finances were not in balance. Further, we found that 
level of position or job title has its own difficulties, particularly for women in those professions that have 
traditionally had few if any "upward" career paths open to them. This argues that judgments of effectiveness 
arc perhaps best made by expert judges who judge behavior in context through interviews. This has its own 
difficulties; however, it seems to pay off in generating abilities that do discriminate effective alumna 
performance. 

• How make assessment cost effective? What legal issues are involved in performance samples of 
employees for post-college studies? 

Clearly, costs for such an effort have still to be determined. Nor can we overlook the legal issues that are 
likely to arise in collaborative studies where colleges study their alumnae at work, particularly if alumnae 
contribute work samples or information from performance appraisals. 

In some of our earlier work, we were advised by our colleagues in the business community that judgments we 
might ask of employers about alumna performance were subject to the regulations for performance appraisal 
information, and that any judgments employers made-even for purposes other than performance 
appraisal— had similar bounds. The oft-suggested comment that employer ratings can be used as criterion 
information for alumna effectiveness, runs up against that concern, along with concern for comparability of 
ratings across judges. 

In sum, we believe that there is enough evidence to suggest that performance interviews can yield the kind of 
information that will enable complex, integrated performance to be assessed at work and in citizenship roles. 
A good deal of effort will need to accompany such an assessment on a wider scale. 



Principle Learned #7. Faculty, professionals, and employers will invest in understanding the relationship 
between education and work if they can create contextuaily rich descriptions of performance in relation 
to their judgment of what abilities to develop. 



We expect that the evidence that has accumulated in this paper will demonstrate that faculty, professionals, 
and employers will invest in a national assessment system because they will come face to face with 
descriptions of performance that illuminate their educational mission and help develop assessment techniques. 
Therefore, we recommend the following: 



Recommendation #7. Faculty, employers, and public 
representatives should be involved in making judgments 
about the abilities essential to effective post-college 
performance at work and in citizenship roles. 



IMPLICATIONS, ISSUES, AND QUESTIONS 

Here we repeat an implication that has had major play in Alverno's perspective on institutional assessment: 
that assessment means creating a context where everyone's investment leads to educational improvement, and 
that the processes that involve the key players are as important as the method itself (Loacker, 1991; 
Mentkowski, 1991). 
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Since we have developed that idea elsewhere, we will close with an issue that we believe underlies much of 
what we have said here. 

• How assess for what employers and the public really want to see: 
personal maturity, leadership, ethical decision-making, and integrated performance? 

The abilities we have described and defined in this paper should lead the reader to understand that much of 
what faculty and students, employers and employees believe is important, is difficult to codify and measure. 
While, in all our research, we saw elements of personal maturity, leadership, ethical decision-making, and 
integrated performance, it is clear that these broad outcomes are not tapped in full. To what degree will an 
assessment system yield the kind of information that will enable all of us to draw the conclusion that we are 
closer to seeing these outcomes in college graduates? 

The arguments presented in this paper depend on substantial investment on the part of everyone involved. 
However, the kind of national assessment system we propose also creates increased investment. We need to 
assume that those responsible for the design to implement the national goals are willing to accept available 
data on abilities/outcomes that link education and work, and to pursue the implications of that data. If so, they 
should be able to begin building a national assessment system that would ultimately contribute to the 
improvement of both higher education and life after graduation for the American citizen at work and in other 
roles in society. 
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Abilities that 

• involve the whole person 

• are teachable 
ALVERNO COLLEGE . can be B5SCSSed 

Milwaukee, Wisconsin ♦ transfer across settings 

APPENDIX A: • are continually re-evaluated 

ABILITY-BASED LEARNING PROGRAM and re-deflned 

The curriculum is an ability-based, ou^ome-oriented approach to liberal arts/professional education. To earn a degree at Alverno 
College a student demonstrates the eight broad abilities listed below, at increasingly complex levels, in general educalion and in 
her areas of specialty. 

Tnese abilities constitute liberal education at the college and undergird and infuse advanced study in the disciplines and profes- 
sions. Within the curriculum of a given major, the student develops the abilities according to the distinctive requirements of the 
disciplines and professions. 

Throughout her course of studies, the student participates in performance-based assessments and learns to assess hSTself. Her 
progression toward a degreo is based upon these assessments, both internal and external. 

With demonstrated achievement at each level the student receives one level unit. For a Bachelor's degree, in addition to 32 units 
awarded when she has demonstrated the first four levels of each of the eight abilities, the student must achieve another 8 units, 
at least one of them at level 6. Advanced levels of any given ability require more time and effort to achieve than lower ones. For 
an Associate of Arts degree in General Studies, a student demonstrates her ability at the first four levels in each of the eight 
areas. 

Abilities and Developmental Levels 

1 Develop communication ability (effectively send and respond to communications for varied audiences and 
purposes) _ 

Level 1 — Identify own strengths and, weaknesses as communicator 
Level 2 — Show analytic approach to effective communicating 
Level 3 — Communicate effectively 

Level 4 — Communicate effectively making relationship; out of explicit frameworks from at 

least three major areas of knowledge 
In majors and areas of specialization: 

Level 5 — Communicate effectively, with application of communications theory 
Level 6 — Communicate with habitual effectiveness and application of theory, through 

coordinated use of different media that represent contemporary technological 

advancement in the communications field 



IN WRITING, 
READING, 
SPEAKING, 
LISTENING. 
USING MEDIA, 
QUANTIFIED 
DATA, AND 
THE COMPUTER 



Develop analytical capabilities 

Level 1 — Show observational skills 
Level 2 — Draw reasonable inferences from observations 
Level 3 — Perceive and make relationships 
Level 4 — Analyze structure and organization 
In majors and areas of specialization: 

Level 5 — Establish ability to employ frameworks from area of concentration or support area discipline in order to 
analyze 

Level 6 *~ Master ability to employ independently the frameworks from area of concentration or support area 
discipline in order to analyze 

Develop workable problem-solving skill 
Level 1 — Articulate and evaluate own problem-solving process 

Level 2 — Define problems or design strategies to solve problems using discipline-related frameworks 
Level 3 — Select or design appropriate frameworks and strategies to solve problems 
Level 4 — Implement a solution and evaluate the problenrvsolving process used 
In majors and areas of specialization: 

Level 5 — Design and implement a process for resolving a problem which requires collaboration with others 
ERJC Level 6 — Demonstrate facility in solving problems in a variety of situations 
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DESIGNING A NATIONAL ASSESSMENT SYSTEM: ASSESSING ABILITIES THAT CONNECT EDUCATION AND WORK 



page 



APPENDIX A (continued). 

4 Develop facility in making value judgments and independent decisions 

Level 1 — Identity own values 

Level 2 — Infer and analyze values in artistic and humanistic works 
L eve i 3 — Relate values to scientific and technological development 
Level 4 — Engage in valuing in decisionmaking in multiple contexts 
In majors and areas ot specialization: 

Level 5 — Analyze and formulate the value foundation/framework of a specific area of knowledge, in its theory and 
practice 

Level 6 — Apply own theory of value and the value foundation of an area of knowledge in a professional context 

5 Develop facility for social Interaction 

Level 1 — Identify own interaction behaviors utilized in a group problenvsolving situation 

Level 2 — Analyze behavior of others within two theoretical frameworks 
Level 3 — Evaluate behavior of self within two theoretical frameworks 

Level 4 — Demonstrate effective social interaction behavior in a variety of situations and circumstances 
In majors and areas of specialization: 

Level 5 — Demonstrate eflective interpersonal and intergroup behaviors in cross-cultural interactions 
Level 6 — Facilitate effective interpersonal and intergroup relationships in one's professional situation 

6 Develop global perspectives 

Level 1 — Assess own knowledge and skills to think about and act on global concerns 

Level 2 — Analyze global issues from multiple perspectives 

Level 3 — Articulate understanding of interconnected local and global issues 

Level 4 — Apply frameworks in formulating a response to global concerns and local issues 

In majors and areas of specialization: 

Level 5 — Generate theoretical and pragmatic approaches to global problems, within a disciplinary or 

professional context 
Level 6 — Develop responsibility toward the global environment in others 

7 Develop effective citizenship 

Level 1 — Assess own knowledge and skills in thinking about and acting on local issues 
Level 2 — Analyze community issues and develop strategies for informed response 

Level 3 — Evaluate personal and organizational characteristics, skills and strategies that facilitate accomplishment 
of mutual goals 

Level 4 — Apply her developing citizenship skills in a community setting 
In majors and areas of specialization: 

Level 5 — Show ability to plan for effective change in social or professional areas 
Level 6 — Exercise leadership in addressing social or professional issues 

8 Develop aesthetic responsiveness: Involvement with the arts 

Level 1 — Express response to selected arts in terms of their formal elements and personal background 

Level 2 — Distinguish among artistic forms in terms of their elements and personal response to selected art works 

Level 3 — Relate artistic works to the contexts from which they emerge 

Level 4 — Make and defend judgments about the quality of selected artistic expressions 

In majors and areas of specialization: 

Level 5 — Choose and discuss artistic works which reflect personal vision of what it means to be human 

Level 6 — Demonstrate the impact of the arts on your own life to this point and project their role in personal future 



© Copyright 1973. Revised May, 1991. Alverno College Production?, Milwaukee, Wisconsin. All rights reserved under U.S., 
International and Universal Copyright Conventions. Reproduction in part or whole by any method is prohibited by law. 
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ALVERNO COLLEGE 

Milwaukee, Wl 
Office of Research and Evaluation 

Appendix B: Abilities/outcomes Applied to Five-year Alumnae Interview Data 
(Behavioral Event interview and Perspectives Interview) 

Abilities (Appendix A) 

Communication 
Analysis 
Problem Solving 
Valuing 

Social Interaction 
Global Perspectives 
Effective Citizenship 
Aesthetic Responsiveness 

(Alvcmo College Faculty, 1976, revised 1985) 

Understanding of domains of knowledge: 

Natural Sciences and Mathematics 
Behavioral and Social Sciences 
Arts and Humanities 

Six Performance Characteristics 
Integration 
Independence 
Creativity 
Awareness 
Commitment 
Habituality 

(Assessment Committee/Office of Research and Evaluation, 1983) 

Other Expected Abilities/Outcomes Common to all Student: 10 Abilities that transcend (hose 

developed through study and experience in disciplinary, interdisciplinary and professic 1 areas. Alvemo 
does not assess these directly for awarding academic credit, but includes them as outcomes measured in its 
college outcomes studies. 

Special abilities: In the process of developing the eight abilities mentioned above in the context of various 
areas of knowledge and according to the educational structure and process designed by the faculty, students 
develop the following: 

• Learning how to learn: learning approaches, preferences, styles; relationship of theory to practice; 
experience-its nature, purpose and relationship to learning 

• Self assessment: ability and willingness to engage in diagnostic evaluation of strengths and weaknesses 
based on personal, academic and professional standards of achievement 

• Self assessment assists in "understanding oneself" in relation to work/civic duty, multicultural 
environment. 

• Cooperation/collaboration in community activity— academic, civic, professional 

• Understanding oneself in relationship to world of work/civic duty 

• Understanding oneself in multicultural (environment) context 
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"Perspective! on Education at Alvemo College: An Integrated Liberal/Professional Educational Program," Curriculum Committee, 1985; 
Addendum, 1988; Addendum, 4/27/1990, Addendum, 10/1990. 
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Appendix B (continued): Abilities/outcomes Applied to Five-year Alumnae Interview Data (Behavioral 
Event Interview and Perspectives Interview) 

Other Expected Abilities/Outcomes Common to all Students 11 but developed according to the special 
characteristics of each discipline/profession and reflecting the uniqueness-style, interests, abilities-of each 
student. 

Integration and Synthesis: Major and support areas of study 

Major and humanistic studies (values/ethics; aesthetic works) 
Major and liberal education abilities 

Note: Expectation is that students develop the ability to integrate two significant areas of study; the ability 
to integrate and synthesize is among the most important liberal arts abilities (integrate major and 
support areas; integrate abilities within the disciplines; integrate knowing and doing) 

Development of interests for future pursuit: intellectual; social; artistic 

Personal and professional integrity (integrate one's personal and professional life and value system) 

Use of information systems (use of computer, bibliographic and information rclricval systems); 

develop sense of self as intelligent user of technology and as one who understands the impact of 
technology on personal, family, civic, work, national, global life 

Develop sense of self as intelligent user of technology and as one who understands the impact of 
technology on personal, family, civic, work, national, global life 

Understanding and experience of distinction between in-depth and general knowledge 
Commitment to quality .excellence 



Professional Competences 



Alverno Teaching Study Competences 



Alverno Nursing Study Competences 



Conceptualization 
Diagnosis 
Coordination 
Communication 
Integrative Interaction 



(Alverno College Education 
Department/Office of Research and 
Evaluation, 1979; Dicz, 1990) 



Conceptualizing 
Emotional Stamina 
Ego Strength 
Positive Expectations 
Independence 
Reflective Thinking 
Helping 
Influencing 



Coaching 

(DeBack & Mcntkowski, 1986; 
Mcntkowski, DeBack, Allen & Blanton, 
1980) 



" "Perspectives on Education at Alverno College: An Integrated Liberal/Professional Educational Program," Curriculum Committee, 1985; 
Addendum, 1988; Addendum, 4/27/1990. Addendum, 10/1990. 
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Appendix B (continued): Abilities/outcomes Applied to Five-year Alumnae Interview Data (Behavioral 
Event Interview and Perspectives Interview) 



McBer Management Competences 
Socio-Emotional Maturity 
Self-Control 
Spontaneity 
Perceptual Objectivity 
Accurate Self-Assessment 
Stamina and Adaptability 
Entrepreneurial Abilities 
Efficiency Orientation 
Proactivity 
Intellectual Abilities 
Logical Thought 
Conceptualisation 

Diagnostic Use of 
Concepts 
Interpersonal Abilities 
Development of Others 
Expressed Concern with Impact 
Use of Unilateral Power 
Use of Socialized Power 
Concern with Affiliation 
Positive Regard 
Management of Groups 
Self-Presentation 
Oral Communication 
(McBer and Co., 
1978;Mentkowski, O'Brien, 
McEachem & Fowler, 1982; 
Schall, Guinn, Qualich, Kramp, 
Schmitz & Stewart, 1984) 

Klemp "Generic" Competences 
Cognitive Abilities 

Evaluative Thinking/Diagnostic 
Skill 

Conceptualization 
Systematic Thinking 
Interpersonal Abilities 
Influence Skill 
Sensitivity 

Using the Informal Process 
Intrapersonal Factors 
Confidence 
Initiative 
Persistence 
(Klemp, 1988) 



McBer and Company M Generic M 
Competences 

Achievement Orientation 

Initiative 

Concern for Order and Reduction 

of Uncertainty 
Use of Concepts 
Analytic Thinking 
Pattern Recognition 
Self Confidence 
Self Control 
Tenacity 

Technical Expertise 
Concern with Impact 
Interpersonal Sensitivity 
Persuasiveness 
Developing Others 
Directing Others 
Group Management 
Relationship Building 
(Spencer & Spencer 1988) 



Cross reference: MM-05/31/90 BE10UTCOME/MAROA/IBM38 (updated from 5/31/90) (updated from 11/22/88) 
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Appendix C: Summary of BEI Generic Abilities. 
SOCIO-EMOTIONAL MATURITY 

1. Sclf-Control/E motional Stamina : Appropriately and effectively controls emotions so that they do not disrupt performance. 

2. Ego Strength : Is assertive or perseveres in judgment while withstanding confrontation, disagreement or disapproval. 
Requires element of risk. 

3. Stamina : Perseveres in a potentially or actually stressful situation. 

4. Spontaneity/Curiosity: Playful thought, action or expression. 

5. Perceptual Objectivity : Demonstrates understanding of more than one perspective. 

6. Accurate Self-Assessment : Accurately assesses strengths and weaknesses of own performance to improve or assess own 
abilities. 

7. Reflective Thinking/Valuing: Identifies and reflects upon her own behavior, feelings or beliefs and their consequences. It 
may include reflecting upon a weakness or mistake, and must result in showing or searching for new insight about self or 
values. 

ENTREPRENEURIAL ABILITIES 

8. Proactivitv: Taking action that is not scripted by the situation or her role and taking responsibility for the 
outcome/judgment (not "We"). 

A. Initiative 

B. Risk taking initiative 

9. Achievement Orientation: Desire to do something better than a standard of excellence. 

10. Achievement Actions : Actions toward more efficient use of time and resources. 
INTELLECTUAL ABILITIES 

1 1. Specialized Knowledge : Use of knowledge that takes more than one year to acquire on the job. 

12. Diagnostic Information Seeking : Seeks information in an ambiguous situation. 

13. Reflectively Coordinated Practice: Receptive to information value of ongoing events: coordinates ongoing actions, relates 
unusual observations or uses new information to adjust ongoing actions or plans. 

14. Diagnostic recognition : Comparing a stimulus to an understanding or representation stored in memory. 

15. Lists Alternatives: Sees a range of implications, consequences or alternatives, or sees if-then relationships. 

16. Hypothetical/Causal Thinking : Shows systematic understanding of possibility and causality. 

17. Planning/Systematic Thinking : Takes well-ordered and logical approaches to analyzing problems, organizing work and 
planning action. 

18. Conceptualization : Identifying and seeing the key relationships between key issues while understanding the big picture. 
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Appendix C (continued): Summary of BEI Generic Abilities. 
INTERPERSONAL 

19. Formal Communication (demonstrated at time of performance) : Effectively communicating in formal writing, speaking or 
planned interactive situation. 



20. Concern With Affiliation : Wanting to be with someone else in order io enjoy mutual friendship or company (warmth 
necessary): Exhibits concern over establishing, maintaining or restoring a warm relationship with another person. This 
relationship is most adequately described by the word friendship. 

21. Affiliative Action: Enjoying, maintaining or establishing companionship for its own sake (warmth necessary). 



22. Positive Regard: Showing respect for others: seeing them as capable and worthy. 

23. Sensitive to Individual Pyrenees: Being aware of and responsive to individual differences. 

24. Accurate Empathy: Effectively reads the moods and feelings of others. 



25. Development of Others : Use of a variety of strategies to insure others' development and to improve their performance. 

26. Development of Self : Develops own knowledge, skills or capability. 

27. Concern With Impact: Exhibits concern about establishing, maintaining or restoring impact, control or influence over 
other(s) (beyond routine and not avoiding power). 

28. Use of Informational/Expert Influence: Uses own credentials for knowledge, access to specific information or construction 
of rationale to persuade others, including building support for ideas/objectives. 

29. Developing Or Using Relational Power: Establishes a warm or inclusive relationship (or uses others' desire for one) in 
order to influence behavior of others. 



30. Use of Socialized/Political Power: Effectively working with others to accomplish tasks. 

A. Political Organizational Action 

B. Dyad/Small Group Action 

31. Development of Organizational Power : Develops organizational power through strategic personal contact or shows 
understanding of how organization is functionally structured. 

A. Develops Organizational Power 

B. Understands Functional Structure 



32. Use of Unilateral Power : Gives directions/orders based upon personal authority or rules/procedures to obtain compliant 
behavior. 
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Type of Job 


Competencies Which Are Most Critical at This Level of Job 


1 Executives 


• Achievement 
Orientation 

• Initiative 

• Use of Concepts 

+ • Pattern Recognition 

• Selt-Confidence 

• Concern with 
Personal Impact 


• Interpersonal Sensitivity 

+ • Use of Influence Strategies 
+ • Group Management 

• Relationship Building 

• Flexibility 




• Achievement 
Orientation 

• Initiative 
i • Tenacity 

• Analytical Thinking 

• Use of Concepts 
+ • Self-Control 

• Self-Confidence 

• Concern for Order 


• Interpersonal Sensitivity 

+ • Directing Others 
+ • Relationship Building 
+ • Developing Others 
+ • Flexibility 
+ • Concern with Personal 
Impact 


Individual 
Contributors 


• Achievement 
Orientation 

• Initiative 

• Tenacity 

1 • Analytical Thinking 
1 • Use of Concepts 


• Technical Expertise 

• Self-Confidence 

• Interpersonal Sensitivity 

• Direct Persuasion 

• Concern for Order 



+ = competence added to this level 
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APPENDIX E: JOB COMPETENCE ASSESSMENT PROCESS 



DEFINE 



o 



© 



© 



© 



Performance 
Effectiveness 
Criteria 



IDENTITY 



Criterion 
Sample 



Collect Data 

~F= 

Behavioral Event 
Interviews 



IDENTIFY 



• Job Tasks 

• Job Competency 
Requirements 



Validate 
Competency 
Model 



I 



Applications 



• Hard data: sales, profits, productivity measures 

• Supervisor nominations 

• Peer ratings 

• Subordinate ratings (e.g., managerial styles, morale) 

• Customer ratings 



• Superior performers 

• Average performers 



Panels 



7" 1 r - ! 1 

Surveys Expert Data Observation 

(360°ratings) System Base ■ 

_] _I l I 



Elements of job person has to perform 
Characteristics of people who do the job well: 
"Competency Model" 



Behavioral Event Interviews 
Tests 

Assessment Center ratings 



} Second criterion 
sample 



Selection 
Training 

Professional development 
Performance appraisal 
Succession planning 
Evaluation of training, professional 
development programs 
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Ref: Lyle Spencer, Soft Skill Competencies (Edinburgh: Scottish Council for Research in Education, 1983.) 
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BEGINNING CRITICAL THINKER 



DEVELOPING CRITICAL THINKER 



ADVANCED CRITICAL THINKER 



Systematic Thinking 

(The critical thinker is careful, accurate, and 
coherent in observation! judgment, 
and communication.) 

1. Strives to notice as much as possible in the 
object of her thought (e.g., a statement, a 
book, a work of art, behavior). 

2. Recognizes that there are appropriate sets 
of concepts (specialized vocabulary) for 
describing different kinds of phenomena and 
generally uses that language appropriately in 
communicating her observations. 

3. Distinguishes between what she observes 
and what she says about the object of that 
observation. 

4. Independently identifies and uses 
conceptual frameworks to organize and 
synthesize her understanding of that which 
she has observed and to communicate that 
understanding to others. 

5. Actively seeks to make connections among 
various experiences (different readings, 
different courses, own experience, current 
events) and formulates general patterns 
which help to make sense of these 
experiences. 



Critical Self-Consciousness of the Thinking Process 

(The critical thinker recognizes that all knowledge is 
problematic rather than absolute because culturally 
conditioned assumptions influence our perception of life 
and the significance we attach to it.) 

1. Always seeks to identify the culturally conditioned 
assumptions which influence her perception and 
judgment and the perceptions and judgments of others. 

2. Recognizes that perception and judgment may only be 
valid within a certain context or situation (i.e., as 
opposed to being universally valid). 

3. Specifies the contextual limits within which her 
perception; ?r,d judgments may be valid; demands this 
specificity of others. 

4. Evaluates the perceptions, judgments, and behavior of 
others relative to the'' context (i.e., withholds absolute 
judgment). 

5. Recognizes that experience may have multiple layers 
of meaning; sees beyond the literal meaning; shows 
tolerance for ambiguity, uncertainty, "loose ends." 

6. Is willing to question the assumptions and validity ot 
her own perceptions and knowledge and those of 
others. 

7. Seeks authority (the knowledge and experience of 
others) as a guide to understanding and action, but 
ultimately takes personal responsibility for all of her 
judgments. 



Reaching Beyond the Conceptual Frameworks 
Which Currently Shape One's Consciousness 

(The critical thinker sees knowledge as a endless 
process rather than a finite object; the critical 
thinker, therefore, habitually seeks out different 
experiences as resources for personal and 
societal growth.) 

1. Actively explores complexity instead of 
ignoring or stepping around it. 

2. Sees problems and issues in a broader 
perspective than that of her own culture or 
interest group. 

3. Actively seeks a critical understanding of 
alternate points of view and ways of knowing. 

4. Has the capacity to mediate between different 
points of view and to integrate diversn views. 

5. Makes personal judgments only after 
exan lining multiple contexts and multiple 
perspectives, and, even then, recognizes the 
limits of her judgment. 

6. Works to promote critical thinking in others 
through sensitive communications designed to 
enhance self-consciousness of assumptions. 

7. Deals with issues of human understanding 
(e.g., truth, relativism, world view) as matters 
of personal significance rather than as 
academic exercises or interesting diversions. 
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Roth, R., & Riordan, T. (1984). Behaviors characteristic of the critical thinker. Milwaukee, Wl: Alverno Productions. 
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These behaviors weie developed by Professor James Roth (History) and Professor Timothy Riordan (Philosophy) for use in an interdisciplinary liberal arts course, "Bridging the Cultural (lap," 
at Alvemo College. 
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Appendix G: Example of the coding of Achievement Actions. Positive Regard and Development of 
Organization Power. 
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SITUATION: 

I developed and continue to work with 
a computerized system that measures 
the times various nursing activities 
require, and then totals the times as 
nurses entered the activities. 

When the system actually began to 
operate within the hospital and 
staff, who are users of the system 
from the standpoint of being the ones 
that have to enter the data, and the 
system would not be a good system 
unless staff nurses consistently 
entered data and believe that they 
should be accurate with it. And as 
I recognized that staff nurses 
respected the system and had 
confidence in it and said, "This 
system tells what we are doing," as 
opposed to the system we had used in 
the past that had little support of 
the users, that made me feel that it 
wasn't just that I thought that we 
had done a good Job. That it really 
is a good system and made me feel it 
is a system should be published or it 
should be broadcast. 

As I have been moving along through 
this project of developing this 
computerized data collection system, 
I did keep track of what I did so 
that I would have a lot of minutes of 
meetings. I carefully planned 
meetings so that I always had 
agendas. I worked with a lot of 
different group* of people, and so I 
had a lot of information that I could 
then look at when I thought about 
writing an article and submitting it 
to the international society. 

I had encouragement from a nursing 
administrator and from the head of a 
technical department who said, "This 
should be written up. This is an 
excellent system; let's write it up; 
let's get it out there." Probably 
one of the big factors that led to 
actually doing it would have been the 
head of the technical department who 

SO 
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Achievement Actions 
(10B2 and 10B3) 



Development of 
Organizational Powe 

continued 

(31A1) 
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Appendix G (continued): Example of the coding of Achievement Actions, P ositive Regard and 
Development of Organization Power. 



said that he would be very happy to 
work with roe on It. He has a word 
processor and did a lot of the 
mechanics of putting; the article 
together. He would take a lot of what 
I had and get that into print so that 
we could start looking at it. That 
was certainly a big reason for moving 
ahead with it too. 

THOUGHT/FELT: 

When I began I wanted to tell about 
the system. By the time I had 
completed the article, I wanted other 
people to know what the system was, 
and that was more important to me. 
But when I began, I really wanted to 
tell all the things that I had done. 
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ACTUALLY DID: 7 5 

To Get Star*, d With the Publishing, 76 

I began by looking at all of the 77 

Information that I had that led up to 78 

the system. And then I did an 79 

outline of what I thought others 80 

would want to hear from within the 81 

context of the information that I had 82 

about the system. The article had to 83 

be shortened considerably because I 84 

began with taking all of the data 85 

that related to the various points in 86 

the outline and then pared it down 87 

from there. I knew it had to be 88 

shortened because we had page number 89 

constraints. 90 



To Shorten the Article, I started by 92 

taking Just what, I thought, had to 93 

be included. The really pertinent 94 

information: that if that was missing 95 

it wouldn't make much sense. And I 96 

tried to keep in mind that if you can 97 

tell what it is you are doing, why 98 

you're doing it, what happened when 99 

you did It, and what are the results 100 

that you see. By this time it was 101 

already in pros. I probably had to 102 

do three revisions before I had it to 103 

the right length. Again, I had help 104 

from the head of the technical 105 

department with the revision process, 106 

but primarily again it was looking at 107 

what would be the points that would 108 

0 interest people. I did have to let 109— i 
go of some of what Interested me the c i 110 ! 
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Appendix G (continued): Example of the coding of Achievement Actions. Positive Regard and 
Development of Organization Power. 



most and some of my pet parts that I 
felt were Just absolutely vital to be 
said. But I did make myself look at 
it as if I were a hospital 
administrator. There are people from 
all areas of health care, 
physicians, and I had to 
what would interest them 



a lot of 
cons ider 
as opposed 



to what 
staff. 



was important for the nursing; 
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The head of the technical department 
was very helpful in that he put 
everything on his word processor and 
so as ( would revise things, he would 
bring them back to me for further 
review and revision. And he would 
again pull back the latest copy of 
what I thought we should have. Then 
after I agreed that it was the way I 
wanted to submit It, he did do the 
final preparation of putting it onto 
the big papers that you have to use. 
He just did editing. I did put him 
on as co-author because we had worked 
on the project together. He has 
since published a paper from his 
department's standpoint on the same 
system which he is Just using the 
material that I had developed, and he 
put me in as a co-author. We had 
agreed t o Chat . 

OUTCOME: 

The article that I had submitted to 
an international society that holds 
a conference every three years was 
accepted for publication in the 
proceedings, and then I was also 
asked to present the paper. That was 
very satisfyinf because I felt that 
the system that we have in place Is 
really very sophisticated, and I 
personally have felt tht.t it's 
probably cne of the best ones that 
I've been able to see anywhere. It 
made me feel that other people at 
least felt that it was a good system 
if they wanted me to come in and 
present 1 1 . 

v'hen I presented, I had a great deal 
of positive feedback with people from 
other countries even commenting to me 
personally that they felt it was the 

82 
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best presentation of a five day 166 

conference. As I thought about that, 167 

I really believe that the reason I 168 

got that kind of feedback wasn't 169 

because of my presentation, but 170 

because I did stick to being able to 171 

present something that "This Is how 172 

we did It; this Is what It Is; this 1*3 

Is how we did It; these are the 174 

results," whereas many of the 175 

presentations were done simply from 176 

description of a system as opposed to 177 

being able to say "this Is what It 4 *° 
Is; etc." 

Note: This write-up was coded for 
many abilities In addition to 
efficiency actions, positive regard, 
and development of organizational 
power. For the sake of clarity, 
those codes are not Included on this 
sample . 
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Larson on Mentkowski/2. 

• 

Mentkowskl cltM large amounts of research, much of it 
performed by tha Alverno faculty ana much of it indeed har own 
(of ton conductad in collaboration with othara), in support of . ths 
ganaral argument and of tha spsciflc actiities proposed. Host of 
tha research ah. cites is unfamiliar to ma, and Z can't appraisa 
its cogency* but Alverno has a racord of such thoroughgoing 
application of aasassmant and auch consistent aassssment of 
ssoessment, if you will, that ona is not inclined to ba skeptical 
or her claims about what paopls at Avemo have done and can do. 
Furthermore, tha paper and ita appendices present so many 
conceptual frameworka and subfrsmsworke, at high levela of 
abatraction and ganaral 1 sat ion, that ona finlshss the papar 
guassing that the Alverno people must have thought of all tha 
possible concepts and interconnections of concspts onm migh* 
imagine for describing critical thinking and problam solving as 
enactsd in specific profssslonal sltustions. 

The preferred technique of assessment (many proceduroa era 
Hatwl and dlanuassd mora briafly) appears to be to interview the 
graduate, and/or to analyse a record of a apaciflc incident in 
which the examinee has to dsmonotrete abilities for the assessor 
to analyze (and/or to collect a portfolio in which the graduate 
demonstrates hsr sbilitiss). 1 connect the procedure to other 
procedures for assessing profssslonal competence that I have 
heard about— those for assessing the sbilitiss of medical 
students anS doctora, for lnatance. The technique reliea on the 
etudy of performance in context, and on the analysis of the 
characteristics of performance. The technique does not simply 
test recall through some sort of written instrument, or tsst 
problem aolving ability by preaentlng students with a written 
problem or with problem-poalng materlela (a le Moranta); it 
requires oral or extensive written diacuaaion o£ a naturaliatic 
situation. Tha evaluator, 1 infer, conetructa a profile of the 
performance in contsxt, and evidently determlnea from that 
analysis whethsr the examinee ia "Integrating" and applying her 
learning as expected and deal red. 

My responses to theee proposels, as I understand them (if 
indeed I am correct in underatsnding that they are Dr. 
Mentkowskl' a propoaala and auggeatlona ) , are principally three. 
First, I am not aure I understand that our assessment of 
students' communication, critical thinking, and problem solving 
abilities might bsst occur some time after the student haa 
completed undergraduate work, or that it might beat be carried on 
inthe context of a workplace. (The three phrases naming the 
atrtiltfae to bs eaaeaaed do not apeclfy that they are to be 
assessed in ths workplscs, though the "briefing paper" for 
authors of position papera invited authors to think of abilitiea 
that would be or are enected in the workplace if the euthora ao 
desired.) Asseesmsnt carried on in a workplace may give evidence 
not mainly about a student' a abilitiea as developed in a college 



wkiii u I • lUOt LCTWAIJ WkUCUC ill" 4-&1 i 1 • 80PPI i 



2128808054* 



812022191738i# 8 



Larson on Mentkowskl/3. 



or university, but also about the graduate's on-the-job training 
and experience. To be surs, there is no reason why on-the-job 
experience should not be part of what a graduate brings to ths 
assessment, but if such experience is going to ba included in tha 
prepsration to ba assessed, how far bayond college experience do 
we go? And if we go beyond college experience , are we learning 
how well we in higher education are achieving, generally, our 
educetional goals, or something else (e. g. , how fortunate the 
atudente are in getting positions that offar good workplace 
training)? It's cleer from discussions in the middle of page 3 
that Mentkowski is aware of this problem, but I'm not aura I see 
exactly how she addresses it* 

8econd, givan the complexity of arranging assesamants of 
students with workplace experience, how practical la it to try to 
arrange such assesaments? How dose one reech the people one wants 
to assees, after thsy have acattared following graduation? How 
doea one anil at thair willingness to participate in an 
aeeeaament? What la one esseeeing: performance in e aituation 
created! (artlflcally) by the axamlner, or in one occurring 
naturally? And if one la looking at a natural aituation, bow doea 
one aaaure the eaaantial comparability of incident a and workplace 
eltuetlone, from student to student, so that as investigators we 
can be sure that we are learning, comparable findings about 
examinees— so that, in turn, we can generalize from what we 
learn? For a nationwide study, how can we be sure that the 
inatitutlona whoae graduates we etudy take a similar view of how 
knowledge and ability come together in the workplace? Could we, 
furthermore, train tna numbers of lnveetigator/obeervero w« would 
need in order to get any sort of a nationel sample of the 
performances of college graduatae? Given what Dr. Lenth has 
taught ue about the wide variation in intereet in aaaessment from 
state to at ate, could we hope to get any agreement about 
educational leaders, public and private, in varloua atates, about 
the value of carrying on thle complex proceee? And, given what 
Dr. Meronto has taught us, could we be sure that an initial 
agreement, however hard won, would eurvlve ehlfta in political 
wlnda from atate to state and in tha nation? 

TMrd, how doea the easassment procedure described by Dr. 
Mentkowpkl work, anyway? What kinda of behavior are aelected for 
examination? How are they aelected? Whet exectly is the procedure 
for determining the characterletlca or quality of the performance 
given? Who makes thoae decisions and how are these people trained 
to make them? (In a national aeeeaament, we'd need lota of 
interpreter and lots of people to train the interpretere.) What 
exectly ia looked at in the performance, egeinet whet criteria or 
eveluetlve grid? There are, it would appear, different kinda of 
interviews and atrateglea for interviewing, and I gather thet 
there would need to be, in e large-ecala atudy, eome agreement 
about the kinda of interviewa to be undertaken. And, if we need 
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to do so, can wa naka tha appraisals of tha individuals studlsd 
comparable enough so that wa can raach sons conclusions? Can wa 
train tha paopla who will conduct the interviews and appraise the 
performance? (Msntkowski, to her credit, recognises— p. 47— how 
different backgrounds of different interviewers/analysts might 
lead them to focus on different features of behavior, and 
perceiva different abllitlea.) From the position paper, z find I 
can't really answer these question*, and I need to, before I can 
make a judgment about the value of these proposals to us. 

I could wish that the paper had been ahortar, and more 
sharply f jcuaed to thij last group of questions, instead of being 
quite so well stocked with theories about learning and behavior 
and with citations to writings not familiar to me (nor, I'd 
guess, to most of us) and not easily acceseible to those who 
would want to confirm the assertions made about what those 
citations seem to establish. 

But I'm struck, finally, aa I ponder the assay, by the 
amatgaaea far mo of two "abilities'' that Mentkowskl discusses *n 1 
one that she doee not explore explicitly: I'm lad to ask to what 
extent the college graduate ought to diaplay these abilities, and 
whether our assessment ought to (and can) address them. They are: 

(1) the ability, in Mentkowakl'e words, to take "achievement 
actions" [p. 9]: not simply to solve" problems but alao 
to put the solutions into action through whet might be 

a aeries of related stepa; 

(2) the ability Talso p f 91 to "have a poaitive influence" 
in an organizational context; in effect, to lead or even 
inspire others to work better within an organization; 

(3) tho ability [Mentkowskl doss not nam* It directly, so 
far ae I can sea] to identify or rocoomiga or ooee 
problems— an ability that may need to accompany or 
precede the ability to aolXM. the probleme. (Richard 
Young, tha well-known rhetorician, once remarked that 
knowledge or wisdom begins "with a furrowed brow.") 

The first of these three abllitlea can probably be aoseaaed 
in a performance interview; the second may be difficult to assess 
sxcept in a sort of wide-ranging interview about tha pereon'a 
overall accomplishments in the workplace. The third should be 
open to assessment through interviews or invitations to writing 
(as in writing about a "case" in an administrative setting), end 
it may indeed be one of the abllitlea that we ought moet directly 
to look for in our college graduates. We should consider whether 
these abilities, especially the third, ahould be on our agenda 
for assessment* 
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Review of: 

Marcia Mentkowski: Designing a National Assessment System: Assessing Abilities 
That Connect Education and Work 
By: Ted Marches© and Barbara Wright 

An outstanding piece of work! We learned an enormous amount from this 
paper, which led us to a much deeper appreciation of the richness and 
complexity of the three abilities an£ convinced us of the importance of 
assessing the abilities in context. 

It now seems unthinkable to us to operationalise the three abilities 
strictly in terms of on-campus considerations; wt- need employers and other 
citizens at the table, all the more so because Goal 5 explicitly places these 
abilities in relation to citizenship and workforce development. The paper also 
makes a persuasive case for the need to assess the three abilities in use, and 
this may well mean looking at additional factors like interpersonal and 
"learning-to-learn" skills. 

To all that, we say Bravo! Let's keep this paper close at hand and 
remain open to its implications as we explicate the abilities and ponder their 
assessment. While the recommendation* presented here are ambitious, they are 
also properly sensitive to the complexity of the three abilities and solidly 
backed by research as well as Alverno's extensive experience. By linking 
assessment of the three abilities to both accountability and improvement 
agendas, the paper suggests that one system, appropriately conceived, can, 
serve both masters. If we follow this paper's lead, the effort to generate 
valid judgments will require significant investment — but a' least we'll have 
some assurance that that investment is serving educational ends. 
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This paper was an Impressive collection of information and ideas about 
what should be assessed and how to assess abilities that link education and 
work. I was envious of the personnel, time and funds that must be available 
at Alverno College for assessment research. I was also impressed with the 
innovations, follow-up, and depth of the Alverno effort. 

The paper was detailed with regard to the specifics of what is done at 
this fairly small private college. However, the author wisely distilled a 
number of lessons learned and made some recommendations, based on the research 
conducted, about how the results might be applied to a national assessment of 
higher order skills - among other things. All of this information 
notwithstanding, I came away from the paper with some concerns about the 
applicability of the results citad in the paper to a national assessment 
effort. In this review, I will address some of those concerns. Please keep 
in mind that my concerns are not necessarily with the Alverno model itself, 
but rather, at least at times, with more generic Issues raised 1n the paper. 
In addition, I must point out that many of my concerns were not with what 
should px could be done, but rather with what I feel can realistically be done 
at this point in time. 

As early as the introduction, I was confronted with the Issue of when 
higher order thinking skills are learned. One of the assumptions made 1n this 
paper was that the context for relating education and work includes before, 
during and after college. That being true, one has to continue to wonder 
about where intervention should be pieced or aimed if we are to substantially 
increase the proportion of college graduates who demonstrate advanced thinking 
skills - if, in fact, we ire going to intervene at all rather than just 
report. Hence, I repeat in this review a point I have previously made in 
others - that the purpose of any assessment program must be made clear from 
the outset. Are we to simply report or take a more active role in making the 
desired outcome come about? 

At the risk of being too narrow in perspective, it was my belief that 
the purpose of these papers, reviews, etc. was to look at wavs to address the 
fifth objective under the fifth national education goal delineated by the 
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accomp 1sh fifth objective £11 M JJJJJM^ work . t believe it was 

Targe rXirt 9 ^^ rttTtSM'mlRbl!. «y'ple. here Is 
for realism. 

Th« nrevious ooint also calls into question the seventh assumption made 
The P rev1 J"5 Poini . £ idea of a national assessment 

in the paper - that readers will accept ine aea be ^ 

system rather than a single national test, in an 

f?r ft FJE' t 2 on trai St a d "t f '.t£ lfflutli mlgh? very 
Slf«^S'toSJT!^i?^W national 'ndex in order to meet the 
immediate objective. 

The first recommendation in the paper was that "A national assessment 

SMly! cSunlcate effectively! and solve problems regardless of spec,f,c 
occupation? 

Tho author at one ooint, develops the idea that "abilities that may be 
differentiated dir ng the first two years of college become integrated during 
thl last two years] although how this happens is not clear to us now." This 
the last iwo /"js, J'"'y"». . it cl ^ r tnat however we decide to go about 
is an important finding that m akes t clear ™J J°J V " [ tn near or * af ter 

! SH m ° o ? ffiffMldSltltn the graduate has been 
! itl fSc« fir several years as suggested in this paper is another 
! r ; t ue%elauX what ^practical and how 9 strongly one should attempt to tie 
the assessment of education and work skills. 

Th vfvr te :o^ ?i!n on sho U id b, 

KlWS 8S a ro tate contexts' Here! again I must question 
♦hs oMcMcallty of such a system on a national scale. The author's 

♦hit "effective assessment means linking Information on how 
St come into a program to hw ihey graduate, and relating those changes 
tn the cu£?1cu um S , leads me to wonder about whose curriculum we might be 
Sealing wUh on™ naUonal scale. Will a national scale assessment program be 
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able to assess individuals and provide them with Individual feedback in 
addition to feedback to their Institutions of higher education? Again, that 
would be great, but Is it feasible in light of the charge and the resources 
available? 

The assessment system described in this paper made great use of the 
contention that interpersonal and intellectual skills were equally Important 
for effective performance and that Interpersonal ability is an Important 
outcome of college. Allow me to once again become a bit skeptical. Are we to 
assume that Interpersonal skills are or 1 mar 11 v a college-developed set of 
skills? If so, that would be saying an awful lot for the relatively few years 
spent at college, and would minimize the earlier, much longer, more 
developmental periods of a person's life. Assuming that the contention is 
true, and that we can and do assess both sets of skills together, what would 
be done with such information? Would it be fed back to individuals and/or 
their Institutions on a national scale? 

One of my biggest concerns about applying the Alverno model nationally 
Is its dependence on lengthy, complex interviews. First, as has been pointed 
out often in the past, interview data can be called into question for many 
reasons - lack of standardized administration, variable judgments about what 
is to be recorded and what is not, the subjectivity of data obtained orally in 
a face to face setting - to name a few. Even if some of those shortcomings 
could be dealt with (and some of them can), how in the world could the 
interview technique be applied on a national scale? 

I was also a bit concerned about using experiential reports obtained 
during interviews as a basis for determining how students learn. I guess that 
all of my previous experience with the rigors of the scientific method leaves 
me less than enthusiastic about building a body of data based upon subjective 
reporting. 

The emphasis throughout the paper on the importance of self-sustained 
learning was not only well documented, but also important to life-long goals 
and careers. My concern with this aspect of the paper came in conjunction 
with the fourth recommendation which stated that H a national assessment system 
should assess self-sustained learning and should Include feedback and 
opportunities to self -assess." I return again to the issue of what is 
practical on a national scale. Can we realistically expect a national 
assessment to provide individual feedback? Further, 1s it realistic to expect 
that we will be able to nationally track graduates in their occupations to 
determine both an improvement and accountability agenda? And if we did and we 
found deficiencies, who should be held accountable for them? Where would we 
go to remedy the situation for future graduates? 

In summary, I found the Alverno assessment work both interesting and 
detailed. As I have previously stated, my concerns are with the practicality 
of the approach on a national scale. While I have already dealt with a number 
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of specific Issues from the paper, I would like to close this review by 
summarizing a few generic concerns that this paper brought to mind. 

1. The purpose and scope of the task at hand were, in my opinion, much 
more limited than the Alverno model would consider. Throughout this review I 
have hearkened back repeatedly to what I saw as the charge - Objective 5 of 
Goal 5! If that 1s not the charge, then the fault lies In my misperceptlon. 

2. A national assessment of higher order thinking will almost of 
necessity have to be summatlve in nature and all terms will have to be 
carefully defined operationally. In other words, if this assessment effort 1s 
funded and supported like other such efforts I am familiar with, we will be 
very limited in what can be done. Tracking students in their jobs, extensive 
interviewing, individual feedback, etc., will simply not be possible. It is 
likely that a single assessment at or immediately after college graduation 
will be what occurs and this would not fit the Alverno approach at all. 

3. A national assessment effort would likely have to be done on a 
sampling basis across states and that would not make Individual feedback to 
students and institutions feasible. Then, too, it would have to be decided 
what the assessment's purpose is. Would it be simply to report national 
results (like the NAEP reports) or would the results be used to promulgate 
change/improvements to the educational establishment? How would the 
individual states react to such an effort - unless of course they had agreed 
to it beforehand - and how likely is that? 

.4. Regardless of the purpose of the assessment effort, the first year or 
two of "assttssment -would have to be done simply to establish baseline data. 
How else would we know if we are "substantially Increasing" the proportion of 
college graduates with advanced higher order skills? 

5. How applicable are the overall results of the Alverno research to a 
national assessment of higher order thinking skills? Since the persons used 
in the study were primarily, if not exclusively, women, how much can we 
generalize to the college population at large? There 1s a growing body of 
literature which Indicates that males and females process Information 
differently and 1t has long been established that females are much more adept 
at verbal activities than males. Hence, 1s it possible that males would not 
Interview or reflect in the same way that the females from Alverno do? I 
would feel a lot better about the findings and recommendations of the paper if 
males had been sampled in numbers proportional to the U.S. college population. 
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Response to Reviewers of 
DESIGNING A NATIONAL ASSESSMENT SYSTEM: 
ASSESSING ABILITIES THAT CONNECT EDUCATION AND WORK 

Marcia Mentkowski 

Alverno College 
Milwaukee, Wisconsin 

Thank you, reviewers, (Ted Marchese, Barbara Wright, Ronald Swanson, Richard Larson) for your careful and 
thoughtful reviews. Two of the reviewers, Richard Larson and Ronald Swanson, stated that a national 
assessment system should focus its attention on assessing college students as a first priority. I agree, and my 
paper's companion piece (Designing a National Assessment System: Alverno' s Institutional Perspective by 
Georgine Loacker) develops that argument. Swanson argues that following up graduates and researching 
abilities in the workplace is somewhat off the main charge from the National Goals Panel. I understand the 
general question to be, "Given that a national assessment system will focus primarily on assessing the college 
student, what is the rationale for investing in assessment of graduates after college?" "Can we entertain such 
an investment in light of some of the practicalities of implementation we face, and the press for immediate 
results?" 

Clearly, the primary focus of a national assessment system will be on college student performance. My 
charge, however, was to assume that public expectations and future needs will not be met with such a singular 
focus, and to consider the broader sweep and context for developing a national assessment system that puts the 
relationship between college learning and work front and center. 

A major source of criticism from employers is that college students are not performing adequately. Swanson 
may be right, that we can assume that an improvement in higher order thinking skills would defacto enable 
graduates to perform better on the job. As an educator and researcher, I can point to some evidence that he is 
right. I cite it often. But as I see it, the public is not buying the kind of evidence we have so far for the links 
between education and work. We have lost the public trust. 

I agree that our first obligation is to improve the system, but improve it how? Once we start, a myriad of 
questions will arise about what should be assessed, who will set standards, and which standards will be 
acceptable both to higher education and employers, such that each would make a considerable investment in 
the activity. 

In my experience, the accountability and improvement purposes of an assessment system are not met in the 
public eye unless the system generates information that considers what the workplace thinks graduates need to 
be prepared for, and whether their bottom line is met— improved performance in their arena. Employers expect 
that the standards for performance that colleges set and that arc illuminated through the National Goals are 
ones that will really make a difference after college. Further, the public will, I believe, buy in when they are 
confronted with standards that are set in part by what is needed now. Higher education will buy in when a 
system generates information that allows them a look at future needs and demands so they can do what they 
think they are best at-preparing students with abilities that last a lifetime. 

Swanson hoped for a less complex, and therefore, more practical approach. Me too. Frankly, I would never 
have guessed in the early design of an institutional assessment system that the dual purposes of accountability 
and improvement meant what they eventually did. They meant following graduates to take a serious look at 
their performance, and investing in research on the abilities of outstanding performers that would help faculty 
refine what to teach for and make them confident fiat their teaching is future oriented and not confined by 
present standards. 
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I learned quickly that developing the public trust is essential tor this institution (wiui no endowment, with 
students who are generally first generation college students on financial aid, with over 20% minority, with 
tuition kept low-$6,390 anoually-to compete with public institutions) especially since most of our graduates 
work right here in the community! Frankly, taking up the issues of relating education to work by following 
up graduates and researching abilities at work was one of the most practical decisions we made. Was it 
feasible? No, not at first, it was not even our first priority. Ultimately, however, we had to deal with these 
issues. Will a national assessment system experience demands for such public accountability? If I read 
America 2Q00 f s tone and tension, it is precisely these issues that are at the heart of an effort to reclaim the 
public trust and improve higher education's ability to deliver. 

Finally, Larson would have appreciated answers to more of the specifics. He wondered why I spent so much 
lime (and space) developing theories of learning and behavior, and not more on giving the specifics of how 
the assessment and research ideas would actually work in a national assessment system. That is the next step, 
once our educational assumptions and theories about student learning and behavior are clarified in a conceptual 
framework for an assessment design. In my experience, it is precisely the assumptions about how people learn 
and how they integrate performance in complex situations that are critical to designing an assessment system. 
An assessment system is a means to an end; it serves educational purposes. Any assessment system that 
serves the 21st century will need to consider the most advanced ideas about how students learn (and we're not 
there yet!), if only to meet the increased demands for more efficient, long-lasting, and meaningful instructional 
and assessment processes. 

A national assessment system will need to be educationally and theoretically grounded in how students learn 
and develop. First, a conceptual framework and a broad design must be in place that reflects the shifts in 
assumptions we care about (e.g., to assess what students know and can do) and the sweep of issues and 
concerns we face. Then, we can, as a group, take up issues surrounding implementation and the more 
exacting procedures that flow from the design. We can take up the insightful questions Larson raised that 
would enable us to get a better handle on just how such a system would work. The more specific descriptions 
of performance interviewing and analysis Alvemo and others have prepared, that are cited, are available. But 
our experience suggests that it is critical to invite our colleagues from the business and professional 
community who are experienced in various forms of assessment in the workplace to join in the activity. 

I certainly appreciate the care and consideration that all the reviewers gave to my paper. I will consult the 
feedback frequently as we continue to work on these issues here at Alverno and will factor it in during the 
many situations where groups of us have an opportunity to work toward the National Goals. 
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